2024-03-25T22:48:04.111 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T22:48:04.111 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/yuriw-2024-03-25_20:36:27-rados-wip-yuri3-testing-2024-03-25-0744-distro-default-smithi/7621635 --name yuriw-2024-03-25_20:36:27-rados-wip-yuri3-testing-2024-03-25-0744-distro-default-smithi --owner scheduled_yuriw@teuthology --description rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/scrub} -- /home/teuthworker/archive/yuriw-2024-03-25_20:36:27-rados-wip-yuri3-testing-2024-03-25-0744-distro-default-smithi/7621635/orig.config.yaml 2024-03-25T22:48:04.135 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T22:48:04.212 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-03-25_20:36:27-rados-wip-yuri3-testing-2024-03-25-0744-distro-default-smithi/7621635 branch: wip-yuri3-testing-2024-03-25-0744 description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/scrub} email: yweinste@redhat.com first_in_suite: false job_id: '7621635' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2024-03-25_20:36:27-rados-wip-yuri3-testing-2024-03-25-0744-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-yuri3-testing-2024-03-25-0744 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 4354a20109fd25703f6611a1a8f196608a256149 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 4354a20109fd25703f6611a1a8f196608a256149 workunit: branch: wip-yuri3-testing-2024-03-25-0744 sha1: 4354a20109fd25703f6611a1a8f196608a256149 owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 6662 sha1: 4354a20109fd25703f6611a1a8f196608a256149 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri3-testing-2024-03-25-0744 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_4354a20109fd25703f6611a1a8f196608a256149/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 4354a20109fd25703f6611a1a8f196608a256149 targets: smithi129.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFcf1ZeqmvRJQbHCZDCLvapqnSlttyxH2G0hT6jN847ksNewhuhchW6nOj65DHNYawsvGdEjmdGdTeO2qwfXlvw= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - scrub teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_20:36:27 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T22:48:04.213 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_4354a20109fd25703f6611a1a8f196608a256149/qa; will attempt to use it 2024-03-25T22:48:04.213 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_4354a20109fd25703f6611a1a8f196608a256149/qa/tasks 2024-03-25T22:48:04.213 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T22:48:04.215 INFO:teuthology.task.internal:Checking packages... 2024-03-25T22:48:04.238 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '4354a20109fd25703f6611a1a8f196608a256149' 2024-03-25T22:48:04.239 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T22:48:04.239 INFO:teuthology.packaging:ref: None 2024-03-25T22:48:04.239 INFO:teuthology.packaging:tag: None 2024-03-25T22:48:04.239 INFO:teuthology.packaging:branch: wip-yuri3-testing-2024-03-25-0744 2024-03-25T22:48:04.239 INFO:teuthology.packaging:sha1: 4354a20109fd25703f6611a1a8f196608a256149 2024-03-25T22:48:04.239 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-yuri3-testing-2024-03-25-0744 2024-03-25T22:48:04.457 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2416-g4354a201-1jammy 2024-03-25T22:48:04.458 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T22:48:04.466 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T22:48:04.466 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T22:48:04.474 INFO:teuthology.task.internal:Saving configuration 2024-03-25T22:48:04.484 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T22:48:04.491 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T22:48:04.516 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi129.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-25_20:36:27-rados-wip-yuri3-testing-2024-03-25-0744-distro-default-smithi/7621635', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 22:42:15.941866', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIP1eqXdAnPWusxRc9BYXMCizroFOYzDC4uX95qRrwBfc'} 2024-03-25T22:48:04.516 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T22:48:04.525 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-03-25T22:48:04.525 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T22:48:04.569 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f844aea4f70>, signals=[15]) 2024-03-25T22:48:04.569 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T22:48:04.576 INFO:teuthology.task.internal:Opening connections... 2024-03-25T22:48:04.577 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi129.front.sepia.ceph.com 2024-03-25T22:48:04.578 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T22:48:04.658 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T22:48:04.665 DEBUG:teuthology.orchestra.run.smithi129:> uname -m 2024-03-25T22:48:04.751 INFO:teuthology.orchestra.run.smithi129.stdout:x86_64 2024-03-25T22:48:04.751 DEBUG:teuthology.orchestra.run.smithi129:> cat /etc/os-release 2024-03-25T22:48:04.798 INFO:teuthology.orchestra.run.smithi129.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-03-25T22:48:04.798 INFO:teuthology.orchestra.run.smithi129.stdout:NAME="Ubuntu" 2024-03-25T22:48:04.798 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION_ID="22.04" 2024-03-25T22:48:04.798 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-03-25T22:48:04.798 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION_CODENAME=jammy 2024-03-25T22:48:04.798 INFO:teuthology.orchestra.run.smithi129.stdout:ID=ubuntu 2024-03-25T22:48:04.798 INFO:teuthology.orchestra.run.smithi129.stdout:ID_LIKE=debian 2024-03-25T22:48:04.798 INFO:teuthology.orchestra.run.smithi129.stdout:HOME_URL="https://www.ubuntu.com/" 2024-03-25T22:48:04.799 INFO:teuthology.orchestra.run.smithi129.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-03-25T22:48:04.799 INFO:teuthology.orchestra.run.smithi129.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-03-25T22:48:04.799 INFO:teuthology.orchestra.run.smithi129.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-03-25T22:48:04.799 INFO:teuthology.orchestra.run.smithi129.stdout:UBUNTU_CODENAME=jammy 2024-03-25T22:48:04.799 INFO:teuthology.lock.ops:Updating smithi129.front.sepia.ceph.com on lock server 2024-03-25T22:48:04.819 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T22:48:04.830 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T22:48:04.857 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T22:48:04.857 DEBUG:teuthology.orchestra.run.smithi129:> test '!' -e /home/ubuntu/cephtest 2024-03-25T22:48:04.861 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T22:48:04.871 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T22:48:04.871 DEBUG:teuthology.orchestra.run.smithi129:> test -z $(ls -A /var/lib/ceph) 2024-03-25T22:48:04.922 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T22:48:04.959 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T22:48:04.978 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T22:48:04.978 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T22:48:04.978 DEBUG:teuthology.orchestra.run.smithi129:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T22:48:04.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T22:48:04.983 DEBUG:teuthology.orchestra.run.smithi129:> uname -r 2024-03-25T22:48:05.030 INFO:teuthology.orchestra.run.smithi129.stdout:5.15.0-56-generic 2024-03-25T22:48:05.030 INFO:teuthology.task.kernel:Running kernel on smithi129: 5.15.0-56-generic 2024-03-25T22:48:05.030 DEBUG:teuthology.orchestra.run.smithi129:> sudo apt-get clean 2024-03-25T22:48:05.161 DEBUG:teuthology.orchestra.run.smithi129:> sudo apt-get update 2024-03-25T22:48:05.321 INFO:teuthology.orchestra.run.smithi129.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-25T22:48:05.321 INFO:teuthology.orchestra.run.smithi129.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-03-25T22:48:05.321 INFO:teuthology.orchestra.run.smithi129.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-03-25T22:48:05.413 INFO:teuthology.orchestra.run.smithi129.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-03-25T22:48:05.600 INFO:teuthology.orchestra.run.smithi129.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,286 kB] 2024-03-25T22:48:05.677 INFO:teuthology.orchestra.run.smithi129.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [595 kB] 2024-03-25T22:48:05.713 INFO:teuthology.orchestra.run.smithi129.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [429 kB] 2024-03-25T22:48:05.718 INFO:teuthology.orchestra.run.smithi129.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [230 kB] 2024-03-25T22:48:05.721 INFO:teuthology.orchestra.run.smithi129.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-03-25T22:48:05.721 INFO:teuthology.orchestra.run.smithi129.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,591 kB] 2024-03-25T22:48:05.747 INFO:teuthology.orchestra.run.smithi129.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [33.7 kB] 2024-03-25T22:48:05.748 INFO:teuthology.orchestra.run.smithi129.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [267 kB] 2024-03-25T22:48:05.750 INFO:teuthology.orchestra.run.smithi129.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,505 kB] 2024-03-25T22:48:05.751 INFO:teuthology.orchestra.run.smithi129.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-03-25T22:48:05.751 INFO:teuthology.orchestra.run.smithi129.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [599 kB] 2024-03-25T22:48:05.758 INFO:teuthology.orchestra.run.smithi129.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [851 kB] 2024-03-25T22:48:05.769 INFO:teuthology.orchestra.run.smithi129.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [162 kB] 2024-03-25T22:48:05.771 INFO:teuthology.orchestra.run.smithi129.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-03-25T22:48:05.790 INFO:teuthology.orchestra.run.smithi129.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-03-25T22:48:05.790 INFO:teuthology.orchestra.run.smithi129.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] 2024-03-25T22:48:05.792 INFO:teuthology.orchestra.run.smithi129.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [290 kB] 2024-03-25T22:48:05.793 INFO:teuthology.orchestra.run.smithi129.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,476 B] 2024-03-25T22:48:05.793 INFO:teuthology.orchestra.run.smithi129.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-03-25T22:48:05.797 INFO:teuthology.orchestra.run.smithi129.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-03-25T22:48:05.798 INFO:teuthology.orchestra.run.smithi129.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,628 kB] 2024-03-25T22:48:05.816 INFO:teuthology.orchestra.run.smithi129.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [35.3 kB] 2024-03-25T22:48:05.816 INFO:teuthology.orchestra.run.smithi129.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [273 kB] 2024-03-25T22:48:05.819 INFO:teuthology.orchestra.run.smithi129.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-03-25T22:48:05.819 INFO:teuthology.orchestra.run.smithi129.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,059 kB] 2024-03-25T22:48:05.828 INFO:teuthology.orchestra.run.smithi129.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [697 kB] 2024-03-25T22:48:05.835 INFO:teuthology.orchestra.run.smithi129.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [240 kB] 2024-03-25T22:48:05.837 INFO:teuthology.orchestra.run.smithi129.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-03-25T22:48:05.838 INFO:teuthology.orchestra.run.smithi129.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.1 kB] 2024-03-25T22:48:05.838 INFO:teuthology.orchestra.run.smithi129.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,160 B] 2024-03-25T22:48:05.857 INFO:teuthology.orchestra.run.smithi129.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.1 kB] 2024-03-25T22:48:05.858 INFO:teuthology.orchestra.run.smithi129.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-03-25T22:48:05.858 INFO:teuthology.orchestra.run.smithi129.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-03-25T22:48:05.858 INFO:teuthology.orchestra.run.smithi129.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-03-25T22:48:05.859 INFO:teuthology.orchestra.run.smithi129.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-03-25T22:48:05.859 INFO:teuthology.orchestra.run.smithi129.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-03-25T22:48:05.879 INFO:teuthology.orchestra.run.smithi129.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-03-25T22:48:05.879 INFO:teuthology.orchestra.run.smithi129.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-03-25T22:48:05.880 INFO:teuthology.orchestra.run.smithi129.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-03-25T22:48:05.880 INFO:teuthology.orchestra.run.smithi129.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-03-25T22:48:11.363 INFO:teuthology.orchestra.run.smithi129.stdout:Fetched 12.5 MB in 2s (6,814 kB/s) 2024-03-25T22:48:12.724 INFO:teuthology.orchestra.run.smithi129.stdout:Reading package lists... 2024-03-25T22:48:12.741 DEBUG:teuthology.orchestra.run.smithi129:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-25T22:48:12.804 INFO:teuthology.orchestra.run.smithi129.stdout:Reading package lists... 2024-03-25T22:48:12.982 INFO:teuthology.orchestra.run.smithi129.stdout:Building dependency tree... 2024-03-25T22:48:12.982 INFO:teuthology.orchestra.run.smithi129.stdout:Reading state information... 2024-03-25T22:48:13.132 INFO:teuthology.orchestra.run.smithi129.stdout:The following packages were automatically installed and are no longer required: 2024-03-25T22:48:13.132 INFO:teuthology.orchestra.run.smithi129.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-25T22:48:13.132 INFO:teuthology.orchestra.run.smithi129.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-25T22:48:13.133 INFO:teuthology.orchestra.run.smithi129.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-25T22:48:13.133 INFO:teuthology.orchestra.run.smithi129.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-03-25T22:48:13.133 INFO:teuthology.orchestra.run.smithi129.stdout: python2.7-minimal 2024-03-25T22:48:13.133 INFO:teuthology.orchestra.run.smithi129.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-25T22:48:13.134 INFO:teuthology.orchestra.run.smithi129.stdout:The following additional packages will be installed: 2024-03-25T22:48:13.134 INFO:teuthology.orchestra.run.smithi129.stdout: linux-generic linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-03-25T22:48:13.134 INFO:teuthology.orchestra.run.smithi129.stdout: linux-headers-generic linux-image-5.15.0-101-generic 2024-03-25T22:48:13.134 INFO:teuthology.orchestra.run.smithi129.stdout: linux-modules-5.15.0-101-generic linux-modules-extra-5.15.0-101-generic 2024-03-25T22:48:13.136 INFO:teuthology.orchestra.run.smithi129.stdout:Suggested packages: 2024-03-25T22:48:13.136 INFO:teuthology.orchestra.run.smithi129.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-03-25T22:48:13.136 INFO:teuthology.orchestra.run.smithi129.stdout:Recommended packages: 2024-03-25T22:48:13.136 INFO:teuthology.orchestra.run.smithi129.stdout: thermald 2024-03-25T22:48:13.182 INFO:teuthology.orchestra.run.smithi129.stdout:The following NEW packages will be installed: 2024-03-25T22:48:13.182 INFO:teuthology.orchestra.run.smithi129.stdout: linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-03-25T22:48:13.182 INFO:teuthology.orchestra.run.smithi129.stdout: linux-image-5.15.0-101-generic linux-modules-5.15.0-101-generic 2024-03-25T22:48:13.182 INFO:teuthology.orchestra.run.smithi129.stdout: linux-modules-extra-5.15.0-101-generic 2024-03-25T22:48:13.183 INFO:teuthology.orchestra.run.smithi129.stdout:The following packages will be upgraded: 2024-03-25T22:48:13.183 INFO:teuthology.orchestra.run.smithi129.stdout: linux-generic linux-headers-generic linux-image-generic 2024-03-25T22:48:13.248 INFO:teuthology.orchestra.run.smithi129.stdout:3 upgraded, 5 newly installed, 0 to remove and 298 not upgraded. 2024-03-25T22:48:13.248 INFO:teuthology.orchestra.run.smithi129.stdout:Need to get 113 MB of archives. 2024-03-25T22:48:13.248 INFO:teuthology.orchestra.run.smithi129.stdout:After this operation, 582 MB of additional disk space will be used. 2024-03-25T22:48:13.248 INFO:teuthology.orchestra.run.smithi129.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-101-generic amd64 5.15.0-101.111 [22.6 MB] 2024-03-25T22:48:13.729 INFO:teuthology.orchestra.run.smithi129.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-101-generic amd64 5.15.0-101.111 [11.5 MB] 2024-03-25T22:48:13.834 INFO:teuthology.orchestra.run.smithi129.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-101-generic amd64 5.15.0-101.111 [63.9 MB] 2024-03-25T22:48:14.792 INFO:teuthology.orchestra.run.smithi129.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.101.98 [1,698 B] 2024-03-25T22:48:14.792 INFO:teuthology.orchestra.run.smithi129.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.101.98 [2,486 B] 2024-03-25T22:48:14.792 INFO:teuthology.orchestra.run.smithi129.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101 all 5.15.0-101.111 [12.3 MB] 2024-03-25T22:48:14.933 INFO:teuthology.orchestra.run.smithi129.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101-generic amd64 5.15.0-101.111 [2,876 kB] 2024-03-25T22:48:14.963 INFO:teuthology.orchestra.run.smithi129.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.101.98 [2,332 B] 2024-03-25T22:48:15.354 INFO:teuthology.orchestra.run.smithi129.stdout:Fetched 113 MB in 2s (63.8 MB/s) 2024-03-25T22:48:15.532 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package linux-modules-5.15.0-101-generic. 2024-03-25T22:48:17.626 INFO:teuthology.orchestra.run.smithi129.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-03-25T22:48:17.630 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../0-linux-modules-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-25T22:48:17.754 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-25T22:48:19.888 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package linux-image-5.15.0-101-generic. 2024-03-25T22:48:19.910 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../1-linux-image-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-25T22:48:20.006 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-25T22:48:20.416 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-101-generic. 2024-03-25T22:48:20.426 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-25T22:48:20.457 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-25T22:48:27.208 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../3-linux-generic_5.15.0.101.98_amd64.deb ... 2024-03-25T22:48:27.324 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking linux-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-25T22:48:27.624 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.101.98_amd64.deb ... 2024-03-25T22:48:27.752 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking linux-image-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-25T22:48:28.097 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package linux-headers-5.15.0-101. 2024-03-25T22:48:28.119 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../5-linux-headers-5.15.0-101_5.15.0-101.111_all.deb ... 2024-03-25T22:48:28.163 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-03-25T22:48:34.279 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package linux-headers-5.15.0-101-generic. 2024-03-25T22:48:34.316 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../6-linux-headers-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-25T22:48:34.354 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-25T22:48:37.118 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.101.98_amd64.deb ... 2024-03-25T22:48:37.240 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking linux-headers-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-25T22:48:37.607 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-03-25T22:48:37.728 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-25T22:48:37.863 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up linux-headers-generic (5.15.0.101.98) ... 2024-03-25T22:48:38.000 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-25T22:48:39.319 INFO:teuthology.orchestra.run.smithi129.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-03-25T22:48:39.319 INFO:teuthology.orchestra.run.smithi129.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-03-25T22:48:39.319 INFO:teuthology.orchestra.run.smithi129.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-101-generic 2024-03-25T22:48:39.319 INFO:teuthology.orchestra.run.smithi129.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-101-generic 2024-03-25T22:48:39.477 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-25T22:48:41.562 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-25T22:48:43.013 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up linux-image-generic (5.15.0.101.98) ... 2024-03-25T22:48:43.147 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up linux-generic (5.15.0.101.98) ... 2024-03-25T22:48:43.274 INFO:teuthology.orchestra.run.smithi129.stdout:Processing triggers for linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-25T22:48:43.367 INFO:teuthology.orchestra.run.smithi129.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-03-25T22:48:43.367 INFO:teuthology.orchestra.run.smithi129.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-101-generic 2024-03-25T22:48:58.185 INFO:teuthology.orchestra.run.smithi129.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-03-25T22:48:58.185 INFO:teuthology.orchestra.run.smithi129.stdout:Sourcing file `/etc/default/grub' 2024-03-25T22:48:58.210 INFO:teuthology.orchestra.run.smithi129.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-25T22:48:58.238 INFO:teuthology.orchestra.run.smithi129.stdout:Generating grub configuration file ... 2024-03-25T22:48:58.479 INFO:teuthology.orchestra.run.smithi129.stdout:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-03-25T22:48:58.496 INFO:teuthology.orchestra.run.smithi129.stdout:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-03-25T22:48:58.781 INFO:teuthology.orchestra.run.smithi129.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-25T22:48:58.786 INFO:teuthology.orchestra.run.smithi129.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-25T22:48:58.852 INFO:teuthology.orchestra.run.smithi129.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-25T22:48:58.858 INFO:teuthology.orchestra.run.smithi129.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-25T22:48:59.034 INFO:teuthology.orchestra.run.smithi129.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-25T22:48:59.034 INFO:teuthology.orchestra.run.smithi129.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-03-25T22:48:59.034 INFO:teuthology.orchestra.run.smithi129.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-25T22:48:59.073 INFO:teuthology.orchestra.run.smithi129.stdout:done 2024-03-25T22:48:59.456 DEBUG:teuthology.orchestra.run.smithi129:> dpkg -s linux-image-generic 2024-03-25T22:48:59.477 INFO:teuthology.orchestra.run.smithi129.stdout:Package: linux-image-generic 2024-03-25T22:48:59.477 INFO:teuthology.orchestra.run.smithi129.stdout:Status: install ok installed 2024-03-25T22:48:59.477 INFO:teuthology.orchestra.run.smithi129.stdout:Priority: optional 2024-03-25T22:48:59.477 INFO:teuthology.orchestra.run.smithi129.stdout:Section: kernel 2024-03-25T22:48:59.477 INFO:teuthology.orchestra.run.smithi129.stdout:Installed-Size: 21 2024-03-25T22:48:59.477 INFO:teuthology.orchestra.run.smithi129.stdout:Maintainer: Ubuntu Kernel Team 2024-03-25T22:48:59.477 INFO:teuthology.orchestra.run.smithi129.stdout:Architecture: amd64 2024-03-25T22:48:59.478 INFO:teuthology.orchestra.run.smithi129.stdout:Source: linux-meta 2024-03-25T22:48:59.478 INFO:teuthology.orchestra.run.smithi129.stdout:Version: 5.15.0.101.98 2024-03-25T22:48:59.478 INFO:teuthology.orchestra.run.smithi129.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-03-25T22:48:59.478 INFO:teuthology.orchestra.run.smithi129.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-25T22:48:59.478 INFO:teuthology.orchestra.run.smithi129.stdout:Recommends: thermald 2024-03-25T22:48:59.478 INFO:teuthology.orchestra.run.smithi129.stdout:Description: Generic Linux kernel image 2024-03-25T22:48:59.478 INFO:teuthology.orchestra.run.smithi129.stdout: This package will always depend on the latest generic kernel image 2024-03-25T22:48:59.478 INFO:teuthology.orchestra.run.smithi129.stdout: available. 2024-03-25T22:48:59.478 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-101-generic 2024-03-25T22:48:59.479 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T22:48:59.479 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T22:48:59.479 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T22:48:59.479 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi129.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T22:48:59.479 DEBUG:teuthology.orchestra.run.smithi129:> sudo apt-get clean 2024-03-25T22:48:59.711 DEBUG:teuthology.orchestra.run.smithi129:> sudo apt-get update 2024-03-25T22:48:59.861 INFO:teuthology.orchestra.run.smithi129.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-25T22:48:59.872 INFO:teuthology.orchestra.run.smithi129.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-25T22:48:59.892 INFO:teuthology.orchestra.run.smithi129.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-25T22:48:59.976 INFO:teuthology.orchestra.run.smithi129.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-25T22:49:01.258 INFO:teuthology.orchestra.run.smithi129.stdout:Reading package lists... 2024-03-25T22:49:01.276 DEBUG:teuthology.orchestra.run.smithi129:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-25T22:49:01.335 INFO:teuthology.orchestra.run.smithi129.stdout:Reading package lists... 2024-03-25T22:49:01.531 INFO:teuthology.orchestra.run.smithi129.stdout:Building dependency tree... 2024-03-25T22:49:01.531 INFO:teuthology.orchestra.run.smithi129.stdout:Reading state information... 2024-03-25T22:49:01.703 INFO:teuthology.orchestra.run.smithi129.stdout:linux-image-generic is already the newest version (5.15.0.101.98). 2024-03-25T22:49:01.703 INFO:teuthology.orchestra.run.smithi129.stdout:The following packages were automatically installed and are no longer required: 2024-03-25T22:49:01.703 INFO:teuthology.orchestra.run.smithi129.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-25T22:49:01.704 INFO:teuthology.orchestra.run.smithi129.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-25T22:49:01.704 INFO:teuthology.orchestra.run.smithi129.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-25T22:49:01.704 INFO:teuthology.orchestra.run.smithi129.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-25T22:49:01.704 INFO:teuthology.orchestra.run.smithi129.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-25T22:49:01.704 INFO:teuthology.orchestra.run.smithi129.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-25T22:49:01.742 INFO:teuthology.orchestra.run.smithi129.stdout:0 upgraded, 0 newly installed, 0 to remove and 298 not upgraded. 2024-03-25T22:49:01.743 DEBUG:teuthology.orchestra.run.smithi129:> dpkg -s linux-image-generic 2024-03-25T22:49:01.803 INFO:teuthology.orchestra.run.smithi129.stdout:Package: linux-image-generic 2024-03-25T22:49:01.803 INFO:teuthology.orchestra.run.smithi129.stdout:Status: install ok installed 2024-03-25T22:49:01.803 INFO:teuthology.orchestra.run.smithi129.stdout:Priority: optional 2024-03-25T22:49:01.803 INFO:teuthology.orchestra.run.smithi129.stdout:Section: kernel 2024-03-25T22:49:01.803 INFO:teuthology.orchestra.run.smithi129.stdout:Installed-Size: 21 2024-03-25T22:49:01.803 INFO:teuthology.orchestra.run.smithi129.stdout:Maintainer: Ubuntu Kernel Team 2024-03-25T22:49:01.803 INFO:teuthology.orchestra.run.smithi129.stdout:Architecture: amd64 2024-03-25T22:49:01.803 INFO:teuthology.orchestra.run.smithi129.stdout:Source: linux-meta 2024-03-25T22:49:01.803 INFO:teuthology.orchestra.run.smithi129.stdout:Version: 5.15.0.101.98 2024-03-25T22:49:01.803 INFO:teuthology.orchestra.run.smithi129.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-03-25T22:49:01.803 INFO:teuthology.orchestra.run.smithi129.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-25T22:49:01.803 INFO:teuthology.orchestra.run.smithi129.stdout:Recommends: thermald 2024-03-25T22:49:01.804 INFO:teuthology.orchestra.run.smithi129.stdout:Description: Generic Linux kernel image 2024-03-25T22:49:01.804 INFO:teuthology.orchestra.run.smithi129.stdout: This package will always depend on the latest generic kernel image 2024-03-25T22:49:01.804 INFO:teuthology.orchestra.run.smithi129.stdout: available. 2024-03-25T22:49:01.804 DEBUG:teuthology.orchestra.run.smithi129:> mktemp 2024-03-25T22:49:01.849 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.WN5jORel6e 2024-03-25T22:49:01.849 DEBUG:teuthology.orchestra.run.smithi129:> sudo cp /boot/grub/grub.cfg /tmp/tmp.WN5jORel6e 2024-03-25T22:49:01.906 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.WN5jORel6e 2024-03-25T22:49:02.031 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.WN5jORel6e is 10KB 2024-03-25T22:49:02.043 DEBUG:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.WN5jORel6e 2024-03-25T22:49:02.048 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-03-25T22:49:02.100 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-25T22:49:02.100 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-03-25T22:49:02.101 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-03-25T22:49:02.163 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-101-generic 2024-03-25T22:49:02.163 DEBUG:teuthology.orchestra.run.smithi129:> sudo update-grub 2024-03-25T22:49:03.293 INFO:teuthology.orchestra.run.smithi129.stderr:Sourcing file `/etc/default/grub' 2024-03-25T22:49:03.294 INFO:teuthology.orchestra.run.smithi129.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-25T22:49:03.297 INFO:teuthology.orchestra.run.smithi129.stderr:Generating grub configuration file ... 2024-03-25T22:49:03.499 INFO:teuthology.orchestra.run.smithi129.stderr:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-03-25T22:49:03.505 INFO:teuthology.orchestra.run.smithi129.stderr:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-03-25T22:49:03.738 INFO:teuthology.orchestra.run.smithi129.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-25T22:49:03.743 INFO:teuthology.orchestra.run.smithi129.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-25T22:49:03.784 INFO:teuthology.orchestra.run.smithi129.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-25T22:49:03.789 INFO:teuthology.orchestra.run.smithi129.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-25T22:49:03.905 INFO:teuthology.orchestra.run.smithi129.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-25T22:49:03.905 INFO:teuthology.orchestra.run.smithi129.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-25T22:49:03.905 INFO:teuthology.orchestra.run.smithi129.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-25T22:49:03.925 INFO:teuthology.orchestra.run.smithi129.stderr:done 2024-03-25T22:49:03.934 DEBUG:teuthology.orchestra.run.smithi129:> sudo shutdown -r now 2024-03-25T22:49:33.966 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T22:49:33.967 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-25T22:49:33.969 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T22:49:52.415 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.129 2024-03-25T22:50:01.424 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-25T22:50:01.425 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T22:50:04.478 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.129 2024-03-25T22:50:16.490 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-25T22:50:16.492 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T22:50:47.967 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.129 2024-03-25T22:51:02.982 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-25T22:51:02.984 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T22:51:03.272 DEBUG:teuthology.orchestra.run.smithi129:> true 2024-03-25T22:51:03.859 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-25T22:51:03.859 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-101-generic"... 2024-03-25T22:51:03.859 DEBUG:teuthology.orchestra.run.smithi129:> uname -r 2024-03-25T22:51:03.907 INFO:teuthology.orchestra.run.smithi129.stdout:5.15.0-101-generic 2024-03-25T22:51:03.908 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-101-generic vs 5.15.0-101-generic 2024-03-25T22:51:03.908 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T22:51:03.908 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-03-25T22:51:04.908 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T22:51:04.909 DEBUG:teuthology.orchestra.run.smithi129:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T22:51:04.993 INFO:teuthology.orchestra.run.smithi129.stdout:ttyS1 2024-03-25T22:51:05.021 DEBUG:teuthology.parallel:result is None 2024-03-25T22:51:05.022 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T22:51:05.030 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T22:51:05.030 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T22:51:05.035 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T22:51:05.042 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T22:51:05.053 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T22:51:05.053 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T22:51:05.094 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T22:51:05.100 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T22:51:05.101 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T22:51:05.154 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T22:51:05.162 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T22:51:05.164 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T22:51:05.171 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T22:51:05.171 DEBUG:teuthology.orchestra.run.smithi129:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T22:51:05.220 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T22:51:05.231 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T22:51:05.232 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T22:51:05.268 DEBUG:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T22:51:05.316 DEBUG:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T22:51:05.364 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-25T22:51:05.364 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T22:51:05.423 DEBUG:teuthology.orchestra.run.smithi129:> sudo service rsyslog restart 2024-03-25T22:51:05.500 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T22:51:05.508 INFO:teuthology.task.internal:Starting timer... 2024-03-25T22:51:05.508 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T22:51:05.520 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T22:51:05.531 DEBUG:teuthology.task.selinux:Excluding smithi129: OS 'ubuntu' does not support SELinux 2024-03-25T22:51:05.531 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T22:51:05.531 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-03-25T22:51:05.531 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T22:51:05.531 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T22:51:05.543 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-25T22:51:05.549 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-25T22:51:05.688 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T22:51:05.705 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T22:51:05.707 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi129.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T22:58:02.914 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi129.front.sepia.ceph.com')] 2024-03-25T22:58:02.915 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-25T22:58:02.917 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T22:58:02.999 DEBUG:teuthology.orchestra.run.smithi129:> true 2024-03-25T22:58:03.074 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-25T22:58:03.074 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T22:58:03.085 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T22:58:03.085 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T22:58:03.086 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T22:58:03.142 INFO:teuthology.orchestra.run.smithi129.stdout:25 Mar 22:58:03 ntpd[17169]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-03-25T22:58:03.143 INFO:teuthology.orchestra.run.smithi129.stdout:25 Mar 22:58:03 ntpd[17169]: Command line: ntpd -gq 2024-03-25T22:58:03.143 INFO:teuthology.orchestra.run.smithi129.stdout:25 Mar 22:58:03 ntpd[17169]: ---------------------------------------------------- 2024-03-25T22:58:03.143 INFO:teuthology.orchestra.run.smithi129.stdout:25 Mar 22:58:03 ntpd[17169]: ntp-4 is maintained by Network Time Foundation, 2024-03-25T22:58:03.143 INFO:teuthology.orchestra.run.smithi129.stdout:25 Mar 22:58:03 ntpd[17169]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-03-25T22:58:03.143 INFO:teuthology.orchestra.run.smithi129.stdout:25 Mar 22:58:03 ntpd[17169]: corporation. Support and training for ntp-4 are 2024-03-25T22:58:03.143 INFO:teuthology.orchestra.run.smithi129.stdout:25 Mar 22:58:03 ntpd[17169]: available at https://www.nwtime.org/support 2024-03-25T22:58:03.143 INFO:teuthology.orchestra.run.smithi129.stdout:25 Mar 22:58:03 ntpd[17169]: ---------------------------------------------------- 2024-03-25T22:58:03.143 INFO:teuthology.orchestra.run.smithi129.stdout:25 Mar 22:58:03 ntpd[17169]: proto: precision = 0.043 usec (-24) 2024-03-25T22:58:03.144 INFO:teuthology.orchestra.run.smithi129.stdout:25 Mar 22:58:03 ntpd[17169]: basedate set to 2022-02-04 2024-03-25T22:58:03.144 INFO:teuthology.orchestra.run.smithi129.stdout:25 Mar 22:58:03 ntpd[17169]: gps base set to 2022-02-06 (week 2196) 2024-03-25T22:58:03.144 INFO:teuthology.orchestra.run.smithi129.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-25T22:58:03.144 INFO:teuthology.orchestra.run.smithi129.stdout:25 Mar 22:58:03 ntpd[17169]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-25T22:58:03.145 INFO:teuthology.orchestra.run.smithi129.stdout:25 Mar 22:58:03 ntpd[17169]: restrict ::: KOD does nothing without LIMITED. 2024-03-25T22:58:03.145 INFO:teuthology.orchestra.run.smithi129.stderr:restrict ::: KOD does nothing without LIMITED. 2024-03-25T22:58:03.146 INFO:teuthology.orchestra.run.smithi129.stdout:25 Mar 22:58:03 ntpd[17169]: Listen and drop on 0 v6wildcard [::]:123 2024-03-25T22:58:03.146 INFO:teuthology.orchestra.run.smithi129.stdout:25 Mar 22:58:03 ntpd[17169]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-03-25T22:58:03.146 INFO:teuthology.orchestra.run.smithi129.stdout:25 Mar 22:58:03 ntpd[17169]: Listen normally on 2 lo 127.0.0.1:123 2024-03-25T22:58:03.146 INFO:teuthology.orchestra.run.smithi129.stdout:25 Mar 22:58:03 ntpd[17169]: Listen normally on 3 enp3s0f1 172.21.15.129:123 2024-03-25T22:58:03.146 INFO:teuthology.orchestra.run.smithi129.stdout:25 Mar 22:58:03 ntpd[17169]: Listen normally on 4 lo [::1]:123 2024-03-25T22:58:03.146 INFO:teuthology.orchestra.run.smithi129.stdout:25 Mar 22:58:03 ntpd[17169]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7261%5]:123 2024-03-25T22:58:03.146 INFO:teuthology.orchestra.run.smithi129.stdout:25 Mar 22:58:03 ntpd[17169]: Listening on routing socket on fd #22 for interface updates 2024-03-25T22:58:04.143 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:04 ntpd[17169]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-25T22:58:04.144 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:04 ntpd[17169]: can't open /var/log/ntpstats/rawstats.20240325: Permission denied 2024-03-25T22:58:04.144 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:04 ntpd[17169]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-25T22:58:04.144 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:04 ntpd[17169]: can't open /var/log/ntpstats/rawstats.20240325: Permission denied 2024-03-25T22:58:04.144 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:04 ntpd[17169]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-25T22:58:04.144 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:04 ntpd[17169]: can't open /var/log/ntpstats/rawstats.20240325: Permission denied 2024-03-25T22:58:04.144 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:04 ntpd[17169]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-25T22:58:04.144 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:04 ntpd[17169]: can't open /var/log/ntpstats/rawstats.20240325: Permission denied 2024-03-25T22:58:04.144 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:04 ntpd[17169]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-25T22:58:04.144 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:04 ntpd[17169]: can't open /var/log/ntpstats/peerstats.20240325: Permission denied 2024-03-25T22:58:04.144 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:04 ntpd[17169]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-25T22:58:04.144 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:04 ntpd[17169]: can't open /var/log/ntpstats/rawstats.20240325: Permission denied 2024-03-25T22:58:04.144 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:04 ntpd[17169]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-25T22:58:04.144 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:04 ntpd[17169]: can't open /var/log/ntpstats/peerstats.20240325: Permission denied 2024-03-25T22:58:06.144 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:06 ntpd[17169]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-25T22:58:06.144 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:06 ntpd[17169]: can't open /var/log/ntpstats/rawstats.20240325: Permission denied 2024-03-25T22:58:06.144 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:06 ntpd[17169]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-25T22:58:06.144 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:06 ntpd[17169]: can't open /var/log/ntpstats/rawstats.20240325: Permission denied 2024-03-25T22:58:06.144 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:06 ntpd[17169]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-25T22:58:06.145 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:06 ntpd[17169]: can't open /var/log/ntpstats/rawstats.20240325: Permission denied 2024-03-25T22:58:06.145 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:06 ntpd[17169]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-25T22:58:06.145 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:06 ntpd[17169]: can't open /var/log/ntpstats/rawstats.20240325: Permission denied 2024-03-25T22:58:06.145 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:06 ntpd[17169]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-25T22:58:06.145 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:06 ntpd[17169]: can't open /var/log/ntpstats/peerstats.20240325: Permission denied 2024-03-25T22:58:06.145 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:06 ntpd[17169]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-25T22:58:06.145 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:06 ntpd[17169]: can't open /var/log/ntpstats/rawstats.20240325: Permission denied 2024-03-25T22:58:06.145 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:06 ntpd[17169]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-25T22:58:06.145 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:06 ntpd[17169]: can't open /var/log/ntpstats/peerstats.20240325: Permission denied 2024-03-25T22:58:06.145 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:06 ntpd[17169]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-25T22:58:06.145 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:06 ntpd[17169]: can't open /var/log/ntpstats/rawstats.20240325: Permission denied 2024-03-25T22:58:06.145 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:06 ntpd[17169]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-25T22:58:06.145 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:06 ntpd[17169]: can't open /var/log/ntpstats/peerstats.20240325: Permission denied 2024-03-25T22:58:08.144 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:08 ntpd[17169]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-25T22:58:08.144 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:08 ntpd[17169]: can't open /var/log/ntpstats/rawstats.20240325: Permission denied 2024-03-25T22:58:08.145 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:08 ntpd[17169]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-25T22:58:08.145 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:08 ntpd[17169]: can't open /var/log/ntpstats/rawstats.20240325: Permission denied 2024-03-25T22:58:08.145 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:08 ntpd[17169]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-25T22:58:08.145 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:08 ntpd[17169]: can't open /var/log/ntpstats/rawstats.20240325: Permission denied 2024-03-25T22:58:08.145 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:08 ntpd[17169]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-25T22:58:08.145 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:08 ntpd[17169]: can't open /var/log/ntpstats/rawstats.20240325: Permission denied 2024-03-25T22:58:08.145 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:08 ntpd[17169]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-25T22:58:08.145 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:08 ntpd[17169]: can't open /var/log/ntpstats/peerstats.20240325: Permission denied 2024-03-25T22:58:08.145 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:08 ntpd[17169]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-25T22:58:08.145 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:08 ntpd[17169]: can't open /var/log/ntpstats/rawstats.20240325: Permission denied 2024-03-25T22:58:08.145 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:08 ntpd[17169]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-25T22:58:08.145 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:08 ntpd[17169]: can't open /var/log/ntpstats/peerstats.20240325: Permission denied 2024-03-25T22:58:08.145 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:08 ntpd[17169]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-25T22:58:08.145 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:08 ntpd[17169]: can't open /var/log/ntpstats/rawstats.20240325: Permission denied 2024-03-25T22:58:08.146 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:08 ntpd[17169]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-25T22:58:08.146 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:08 ntpd[17169]: can't open /var/log/ntpstats/peerstats.20240325: Permission denied 2024-03-25T22:58:10.144 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:10 ntpd[17169]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-25T22:58:10.144 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:10 ntpd[17169]: can't open /var/log/ntpstats/rawstats.20240325: Permission denied 2024-03-25T22:58:10.144 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:10 ntpd[17169]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-25T22:58:10.144 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:10 ntpd[17169]: can't open /var/log/ntpstats/rawstats.20240325: Permission denied 2024-03-25T22:58:10.144 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:10 ntpd[17169]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-25T22:58:10.145 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:10 ntpd[17169]: can't open /var/log/ntpstats/rawstats.20240325: Permission denied 2024-03-25T22:58:10.145 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:10 ntpd[17169]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-25T22:58:10.145 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:10 ntpd[17169]: can't open /var/log/ntpstats/rawstats.20240325: Permission denied 2024-03-25T22:58:10.145 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:10 ntpd[17169]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-25T22:58:10.145 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:10 ntpd[17169]: can't open /var/log/ntpstats/peerstats.20240325: Permission denied 2024-03-25T22:58:10.145 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:10 ntpd[17169]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-25T22:58:10.145 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:10 ntpd[17169]: can't open /var/log/ntpstats/rawstats.20240325: Permission denied 2024-03-25T22:58:10.145 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:10 ntpd[17169]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-25T22:58:10.145 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:10 ntpd[17169]: can't open /var/log/ntpstats/peerstats.20240325: Permission denied 2024-03-25T22:58:10.145 INFO:teuthology.orchestra.run.smithi129.stdout:25 Mar 22:58:10 ntpd[17169]: ntpd: time slew -0.001576 s 2024-03-25T22:58:10.145 INFO:teuthology.orchestra.run.smithi129.stdout:ntpd: time slew -0.001576s 2024-03-25T22:58:10.146 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:10 ntpd[17169]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-03-25T22:58:10.146 INFO:teuthology.orchestra.run.smithi129.stderr:25 Mar 22:58:10 ntpd[17169]: can't open /var/log/ntpstats/loopstats.20240325: Permission denied 2024-03-25T22:58:10.190 INFO:teuthology.orchestra.run.smithi129.stdout: remote refid st t when poll reach delay offset jitter 2024-03-25T22:58:10.190 INFO:teuthology.orchestra.run.smithi129.stdout:============================================================================== 2024-03-25T22:58:10.190 INFO:teuthology.orchestra.run.smithi129.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-25T22:58:10.190 INFO:teuthology.orchestra.run.smithi129.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-25T22:58:10.190 INFO:teuthology.orchestra.run.smithi129.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-25T22:58:10.190 INFO:teuthology.orchestra.run.smithi129.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-25T22:58:10.190 INFO:teuthology.run_tasks:Running task install... 2024-03-25T22:58:10.198 DEBUG:teuthology.task.install:project ceph 2024-03-25T22:58:10.199 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '4354a20109fd25703f6611a1a8f196608a256149'}} 2024-03-25T22:58:10.199 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '4354a20109fd25703f6611a1a8f196608a256149'} 2024-03-25T22:58:10.199 INFO:teuthology.task.install:Using flavor: default 2024-03-25T22:58:10.206 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-25T22:58:10.206 INFO:teuthology.task.install:extra packages: [] 2024-03-25T22:58:10.206 DEBUG:teuthology.orchestra.run.smithi129:> sudo apt-key list | grep Ceph 2024-03-25T22:58:10.291 INFO:teuthology.orchestra.run.smithi129.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-03-25T22:58:10.319 INFO:teuthology.orchestra.run.smithi129.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-03-25T22:58:10.320 INFO:teuthology.orchestra.run.smithi129.stdout:uid [ unknown] Ceph.com (release key) 2024-03-25T22:58:10.320 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-03-25T22:58:10.320 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=4354a20109fd25703f6611a1a8f196608a256149 2024-03-25T22:58:10.474 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-03-25-0744/4354a20109fd25703f6611a1a8f196608a256149/ubuntu/jammy/flavors/default/ 2024-03-25T22:58:10.474 INFO:teuthology.task.install.deb:Package version is 19.0.0-2416-g4354a201-1jammy 2024-03-25T22:58:10.587 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-25T22:58:10.587 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-03-25T22:58:10.601 DEBUG:teuthology.orchestra.run.smithi129:> sudo apt-get update 2024-03-25T22:58:10.773 INFO:teuthology.orchestra.run.smithi129.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-25T22:58:10.773 INFO:teuthology.orchestra.run.smithi129.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-03-25T22:58:10.787 INFO:teuthology.orchestra.run.smithi129.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-03-25T22:58:10.874 INFO:teuthology.orchestra.run.smithi129.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-25T22:58:10.883 INFO:teuthology.orchestra.run.smithi129.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-03-25-0744/4354a20109fd25703f6611a1a8f196608a256149/ubuntu/jammy/flavors/default jammy InRelease 2024-03-25T22:58:10.919 INFO:teuthology.orchestra.run.smithi129.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-03-25-0744/4354a20109fd25703f6611a1a8f196608a256149/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-03-25T22:58:10.935 INFO:teuthology.orchestra.run.smithi129.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,290 kB] 2024-03-25T22:58:10.956 INFO:teuthology.orchestra.run.smithi129.stdout:Ign:8 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-03-25-0744/4354a20109fd25703f6611a1a8f196608a256149/ubuntu/jammy/flavors/default jammy Release.gpg 2024-03-25T22:58:10.993 INFO:teuthology.orchestra.run.smithi129.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-03-25-0744/4354a20109fd25703f6611a1a8f196608a256149/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-03-25T22:58:11.035 INFO:teuthology.orchestra.run.smithi129.stdout:Get:10 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-03-25-0744/4354a20109fd25703f6611a1a8f196608a256149/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,405 B] 2024-03-25T22:58:11.037 INFO:teuthology.orchestra.run.smithi129.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [429 kB] 2024-03-25T22:58:11.042 INFO:teuthology.orchestra.run.smithi129.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [230 kB] 2024-03-25T22:58:11.045 INFO:teuthology.orchestra.run.smithi129.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,600 kB] 2024-03-25T22:58:11.067 INFO:teuthology.orchestra.run.smithi129.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [268 kB] 2024-03-25T22:58:11.558 INFO:teuthology.orchestra.run.smithi129.stdout:Fetched 4,075 kB in 1s (5,316 kB/s) 2024-03-25T22:58:12.482 INFO:teuthology.orchestra.run.smithi129.stdout:Reading package lists... 2024-03-25T22:58:12.498 DEBUG:teuthology.orchestra.run.smithi129:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2416-g4354a201-1jammy cephadm=19.0.0-2416-g4354a201-1jammy ceph-mds=19.0.0-2416-g4354a201-1jammy ceph-mgr=19.0.0-2416-g4354a201-1jammy ceph-common=19.0.0-2416-g4354a201-1jammy ceph-fuse=19.0.0-2416-g4354a201-1jammy ceph-test=19.0.0-2416-g4354a201-1jammy ceph-volume=19.0.0-2416-g4354a201-1jammy radosgw=19.0.0-2416-g4354a201-1jammy python3-rados=19.0.0-2416-g4354a201-1jammy python3-rgw=19.0.0-2416-g4354a201-1jammy python3-cephfs=19.0.0-2416-g4354a201-1jammy python3-rbd=19.0.0-2416-g4354a201-1jammy libcephfs2=19.0.0-2416-g4354a201-1jammy libcephfs-dev=19.0.0-2416-g4354a201-1jammy librados2=19.0.0-2416-g4354a201-1jammy librbd1=19.0.0-2416-g4354a201-1jammy rbd-fuse=19.0.0-2416-g4354a201-1jammy 2024-03-25T22:58:12.550 INFO:teuthology.orchestra.run.smithi129.stdout:Reading package lists... 2024-03-25T22:58:12.724 INFO:teuthology.orchestra.run.smithi129.stdout:Building dependency tree... 2024-03-25T22:58:12.724 INFO:teuthology.orchestra.run.smithi129.stdout:Reading state information... 2024-03-25T22:58:12.881 INFO:teuthology.orchestra.run.smithi129.stdout:The following packages were automatically installed and are no longer required: 2024-03-25T22:58:12.881 INFO:teuthology.orchestra.run.smithi129.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-25T22:58:12.881 INFO:teuthology.orchestra.run.smithi129.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-25T22:58:12.881 INFO:teuthology.orchestra.run.smithi129.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-25T22:58:12.881 INFO:teuthology.orchestra.run.smithi129.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-25T22:58:12.881 INFO:teuthology.orchestra.run.smithi129.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-25T22:58:12.882 INFO:teuthology.orchestra.run.smithi129.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-25T22:58:12.883 INFO:teuthology.orchestra.run.smithi129.stdout:The following additional packages will be installed: 2024-03-25T22:58:12.883 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-03-25T22:58:12.883 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-03-25T22:58:12.883 INFO:teuthology.orchestra.run.smithi129.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-25T22:58:12.883 INFO:teuthology.orchestra.run.smithi129.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-03-25T22:58:12.883 INFO:teuthology.orchestra.run.smithi129.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-03-25T22:58:12.883 INFO:teuthology.orchestra.run.smithi129.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-03-25T22:58:12.883 INFO:teuthology.orchestra.run.smithi129.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-03-25T22:58:12.883 INFO:teuthology.orchestra.run.smithi129.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-03-25T22:58:12.883 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-03-25T22:58:12.883 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-03-25T22:58:12.884 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-25T22:58:12.884 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-25T22:58:12.884 INFO:teuthology.orchestra.run.smithi129.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-25T22:58:12.884 INFO:teuthology.orchestra.run.smithi129.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-25T22:58:12.884 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-25T22:58:12.884 INFO:teuthology.orchestra.run.smithi129.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-25T22:58:12.884 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-03-25T22:58:12.884 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-03-25T22:58:12.884 INFO:teuthology.orchestra.run.smithi129.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-03-25T22:58:12.884 INFO:teuthology.orchestra.run.smithi129.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-03-25T22:58:12.884 INFO:teuthology.orchestra.run.smithi129.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-03-25T22:58:12.884 INFO:teuthology.orchestra.run.smithi129.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-03-25T22:58:12.885 INFO:teuthology.orchestra.run.smithi129.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-03-25T22:58:12.885 INFO:teuthology.orchestra.run.smithi129.stdout: xmlstarlet zip 2024-03-25T22:58:12.885 INFO:teuthology.orchestra.run.smithi129.stdout:Suggested packages: 2024-03-25T22:58:12.885 INFO:teuthology.orchestra.run.smithi129.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-03-25T22:58:12.885 INFO:teuthology.orchestra.run.smithi129.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-03-25T22:58:12.885 INFO:teuthology.orchestra.run.smithi129.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-03-25T22:58:12.885 INFO:teuthology.orchestra.run.smithi129.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-03-25T22:58:12.885 INFO:teuthology.orchestra.run.smithi129.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-03-25T22:58:12.885 INFO:teuthology.orchestra.run.smithi129.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-03-25T22:58:12.886 INFO:teuthology.orchestra.run.smithi129.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-03-25T22:58:12.886 INFO:teuthology.orchestra.run.smithi129.stdout:Recommended packages: 2024-03-25T22:58:12.886 INFO:teuthology.orchestra.run.smithi129.stdout: btrfs-tools 2024-03-25T22:58:12.984 INFO:teuthology.orchestra.run.smithi129.stdout:The following NEW packages will be installed: 2024-03-25T22:58:12.985 INFO:teuthology.orchestra.run.smithi129.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-03-25T22:58:12.985 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-03-25T22:58:12.985 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-03-25T22:58:12.985 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-03-25T22:58:12.985 INFO:teuthology.orchestra.run.smithi129.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-03-25T22:58:12.985 INFO:teuthology.orchestra.run.smithi129.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-03-25T22:58:12.987 INFO:teuthology.orchestra.run.smithi129.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-03-25T22:58:12.987 INFO:teuthology.orchestra.run.smithi129.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-03-25T22:58:12.987 INFO:teuthology.orchestra.run.smithi129.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-03-25T22:58:12.987 INFO:teuthology.orchestra.run.smithi129.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-03-25T22:58:12.987 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-03-25T22:58:12.987 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-03-25T22:58:12.987 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-25T22:58:12.987 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-25T22:58:12.987 INFO:teuthology.orchestra.run.smithi129.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-25T22:58:12.987 INFO:teuthology.orchestra.run.smithi129.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-25T22:58:12.987 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-25T22:58:12.988 INFO:teuthology.orchestra.run.smithi129.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-25T22:58:12.988 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-03-25T22:58:12.988 INFO:teuthology.orchestra.run.smithi129.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-03-25T22:58:12.988 INFO:teuthology.orchestra.run.smithi129.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-03-25T22:58:12.988 INFO:teuthology.orchestra.run.smithi129.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-03-25T22:58:12.988 INFO:teuthology.orchestra.run.smithi129.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-03-25T22:58:12.988 INFO:teuthology.orchestra.run.smithi129.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-03-25T22:58:12.988 INFO:teuthology.orchestra.run.smithi129.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-03-25T22:58:12.988 INFO:teuthology.orchestra.run.smithi129.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-03-25T22:58:13.173 INFO:teuthology.orchestra.run.smithi129.stdout:0 upgraded, 118 newly installed, 0 to remove and 285 not upgraded. 2024-03-25T22:58:13.174 INFO:teuthology.orchestra.run.smithi129.stdout:Need to get 178 MB of archives. 2024-03-25T22:58:13.174 INFO:teuthology.orchestra.run.smithi129.stdout:After this operation, 819 MB of additional disk space will be used. 2024-03-25T22:58:13.174 INFO:teuthology.orchestra.run.smithi129.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-03-25T22:58:13.205 INFO:teuthology.orchestra.run.smithi129.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-03-25-0744/4354a20109fd25703f6611a1a8f196608a256149/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2416-g4354a201-1jammy [3,554 kB] 2024-03-25T22:58:13.599 INFO:teuthology.orchestra.run.smithi129.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-03-25T22:58:13.612 INFO:teuthology.orchestra.run.smithi129.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-03-25T22:58:13.693 INFO:teuthology.orchestra.run.smithi129.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-03-25T22:58:13.827 INFO:teuthology.orchestra.run.smithi129.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-03-25-0744/4354a20109fd25703f6611a1a8f196608a256149/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2416-g4354a201-1jammy [3,229 kB] 2024-03-25T22:58:13.922 INFO:teuthology.orchestra.run.smithi129.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-03-25T22:58:13.934 INFO:teuthology.orchestra.run.smithi129.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-03-25T22:58:13.963 INFO:teuthology.orchestra.run.smithi129.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-03-25T22:58:13.973 INFO:teuthology.orchestra.run.smithi129.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-03-25T22:58:13.973 INFO:teuthology.orchestra.run.smithi129.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-03-25T22:58:13.975 INFO:teuthology.orchestra.run.smithi129.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-03-25T22:58:13.993 INFO:teuthology.orchestra.run.smithi129.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-03-25T22:58:14.053 INFO:teuthology.orchestra.run.smithi129.stdout:Get:14 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-03-25-0744/4354a20109fd25703f6611a1a8f196608a256149/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2416-g4354a201-1jammy [712 kB] 2024-03-25T22:58:14.116 INFO:teuthology.orchestra.run.smithi129.stdout:Get:15 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-03-25-0744/4354a20109fd25703f6611a1a8f196608a256149/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2416-g4354a201-1jammy [356 kB] 2024-03-25T22:58:14.124 INFO:teuthology.orchestra.run.smithi129.stdout:Get:16 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-03-25-0744/4354a20109fd25703f6611a1a8f196608a256149/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2416-g4354a201-1jammy [32.8 kB] 2024-03-25T22:58:14.125 INFO:teuthology.orchestra.run.smithi129.stdout:Get:17 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-03-25-0744/4354a20109fd25703f6611a1a8f196608a256149/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2416-g4354a201-1jammy [180 kB] 2024-03-25T22:58:14.154 INFO:teuthology.orchestra.run.smithi129.stdout:Get:18 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-03-25-0744/4354a20109fd25703f6611a1a8f196608a256149/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2416-g4354a201-1jammy [65.7 kB] 2024-03-25T22:58:14.155 INFO:teuthology.orchestra.run.smithi129.stdout:Get:19 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-03-25-0744/4354a20109fd25703f6611a1a8f196608a256149/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2416-g4354a201-1jammy [330 kB] 2024-03-25T22:58:14.162 INFO:teuthology.orchestra.run.smithi129.stdout:Get:20 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-03-25-0744/4354a20109fd25703f6611a1a8f196608a256149/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2416-g4354a201-1jammy [6,660 kB] 2024-03-25T22:58:14.163 INFO:teuthology.orchestra.run.smithi129.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-03-25T22:58:14.507 INFO:teuthology.orchestra.run.smithi129.stdout:Get:22 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-03-25-0744/4354a20109fd25703f6611a1a8f196608a256149/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2416-g4354a201-1jammy [112 kB] 2024-03-25T22:58:14.510 INFO:teuthology.orchestra.run.smithi129.stdout:Get:23 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-03-25-0744/4354a20109fd25703f6611a1a8f196608a256149/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2416-g4354a201-1jammy [466 kB] 2024-03-25T22:58:14.531 INFO:teuthology.orchestra.run.smithi129.stdout:Get:24 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-03-25-0744/4354a20109fd25703f6611a1a8f196608a256149/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2416-g4354a201-1jammy [25.3 MB] 2024-03-25T22:58:14.573 INFO:teuthology.orchestra.run.smithi129.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-03-25T22:58:14.607 INFO:teuthology.orchestra.run.smithi129.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-03-25T22:58:14.610 INFO:teuthology.orchestra.run.smithi129.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-03-25T22:58:14.690 INFO:teuthology.orchestra.run.smithi129.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-03-25T22:58:14.741 INFO:teuthology.orchestra.run.smithi129.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-03-25T22:58:14.766 INFO:teuthology.orchestra.run.smithi129.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-03-25T22:58:14.769 INFO:teuthology.orchestra.run.smithi129.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-03-25T22:58:14.770 INFO:teuthology.orchestra.run.smithi129.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-03-25T22:58:14.782 INFO:teuthology.orchestra.run.smithi129.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-03-25T22:58:14.783 INFO:teuthology.orchestra.run.smithi129.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-03-25T22:58:14.948 INFO:teuthology.orchestra.run.smithi129.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-03-25T22:58:15.032 INFO:teuthology.orchestra.run.smithi129.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-03-25T22:58:15.060 INFO:teuthology.orchestra.run.smithi129.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-03-25T22:58:15.077 INFO:teuthology.orchestra.run.smithi129.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-03-25T22:58:15.122 INFO:teuthology.orchestra.run.smithi129.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-03-25T22:58:15.336 INFO:teuthology.orchestra.run.smithi129.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-03-25T22:58:15.351 INFO:teuthology.orchestra.run.smithi129.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-03-25T22:58:15.358 INFO:teuthology.orchestra.run.smithi129.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-03-25T22:58:15.363 INFO:teuthology.orchestra.run.smithi129.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-03-25T22:58:15.384 INFO:teuthology.orchestra.run.smithi129.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-03-25T22:58:15.388 INFO:teuthology.orchestra.run.smithi129.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-03-25T22:58:15.482 INFO:teuthology.orchestra.run.smithi129.stdout:Get:46 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-03-25-0744/4354a20109fd25703f6611a1a8f196608a256149/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2416-g4354a201-1jammy [5,086 kB] 2024-03-25T22:58:15.555 INFO:teuthology.orchestra.run.smithi129.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-03-25T22:58:15.582 INFO:teuthology.orchestra.run.smithi129.stdout:Get:48 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-03-25-0744/4354a20109fd25703f6611a1a8f196608a256149/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2416-g4354a201-1jammy [244 kB] 2024-03-25T22:58:15.588 INFO:teuthology.orchestra.run.smithi129.stdout:Get:49 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-03-25-0744/4354a20109fd25703f6611a1a8f196608a256149/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2416-g4354a201-1jammy [124 kB] 2024-03-25T22:58:15.594 INFO:teuthology.orchestra.run.smithi129.stdout:Get:50 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-03-25-0744/4354a20109fd25703f6611a1a8f196608a256149/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2416-g4354a201-1jammy [1,526 kB] 2024-03-25T22:58:15.644 INFO:teuthology.orchestra.run.smithi129.stdout:Get:51 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-03-25-0744/4354a20109fd25703f6611a1a8f196608a256149/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2416-g4354a201-1jammy [6,255 kB] 2024-03-25T22:58:15.849 INFO:teuthology.orchestra.run.smithi129.stdout:Get:52 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-03-25-0744/4354a20109fd25703f6611a1a8f196608a256149/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2416-g4354a201-1jammy [22.8 MB] 2024-03-25T22:58:15.870 INFO:teuthology.orchestra.run.smithi129.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-03-25T22:58:15.909 INFO:teuthology.orchestra.run.smithi129.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-03-25T22:58:15.920 INFO:teuthology.orchestra.run.smithi129.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-03-25T22:58:16.125 INFO:teuthology.orchestra.run.smithi129.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-03-25T22:58:16.125 INFO:teuthology.orchestra.run.smithi129.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-03-25T22:58:16.131 INFO:teuthology.orchestra.run.smithi129.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-03-25T22:58:16.137 INFO:teuthology.orchestra.run.smithi129.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-03-25T22:58:16.152 INFO:teuthology.orchestra.run.smithi129.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-03-25T22:58:16.173 INFO:teuthology.orchestra.run.smithi129.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-03-25T22:58:16.179 INFO:teuthology.orchestra.run.smithi129.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-03-25T22:58:16.354 INFO:teuthology.orchestra.run.smithi129.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-03-25T22:58:16.618 INFO:teuthology.orchestra.run.smithi129.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-03-25T22:58:16.686 INFO:teuthology.orchestra.run.smithi129.stdout:Get:65 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-03-25-0744/4354a20109fd25703f6611a1a8f196608a256149/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2416-g4354a201-1jammy [14.0 kB] 2024-03-25T22:58:16.687 INFO:teuthology.orchestra.run.smithi129.stdout:Get:66 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-03-25-0744/4354a20109fd25703f6611a1a8f196608a256149/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2416-g4354a201-1jammy [870 kB] 2024-03-25T22:58:16.704 INFO:teuthology.orchestra.run.smithi129.stdout:Get:67 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-03-25-0744/4354a20109fd25703f6611a1a8f196608a256149/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2416-g4354a201-1jammy [2,434 kB] 2024-03-25T22:58:16.744 INFO:teuthology.orchestra.run.smithi129.stdout:Get:68 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-03-25-0744/4354a20109fd25703f6611a1a8f196608a256149/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2416-g4354a201-1jammy [779 kB] 2024-03-25T22:58:16.761 INFO:teuthology.orchestra.run.smithi129.stdout:Get:69 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-03-25-0744/4354a20109fd25703f6611a1a8f196608a256149/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2416-g4354a201-1jammy [146 kB] 2024-03-25T22:58:16.767 INFO:teuthology.orchestra.run.smithi129.stdout:Get:70 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-03-25-0744/4354a20109fd25703f6611a1a8f196608a256149/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2416-g4354a201-1jammy [2,259 kB] 2024-03-25T22:58:16.796 INFO:teuthology.orchestra.run.smithi129.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-03-25T22:58:16.810 INFO:teuthology.orchestra.run.smithi129.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-03-25T22:58:16.830 INFO:teuthology.orchestra.run.smithi129.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-03-25T22:58:16.843 INFO:teuthology.orchestra.run.smithi129.stdout:Get:74 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-03-25-0744/4354a20109fd25703f6611a1a8f196608a256149/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2416-g4354a201-1jammy [8,625 kB] 2024-03-25T22:58:16.894 INFO:teuthology.orchestra.run.smithi129.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-03-25T22:58:17.120 INFO:teuthology.orchestra.run.smithi129.stdout:Get:76 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-03-25-0744/4354a20109fd25703f6611a1a8f196608a256149/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2416-g4354a201-1jammy [14.2 kB] 2024-03-25T22:58:17.120 INFO:teuthology.orchestra.run.smithi129.stdout:Get:77 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-03-25-0744/4354a20109fd25703f6611a1a8f196608a256149/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2416-g4354a201-1jammy [50.3 MB] 2024-03-25T22:58:17.182 INFO:teuthology.orchestra.run.smithi129.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-03-25T22:58:17.184 INFO:teuthology.orchestra.run.smithi129.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-03-25T22:58:17.187 INFO:teuthology.orchestra.run.smithi129.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-03-25T22:58:17.188 INFO:teuthology.orchestra.run.smithi129.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-03-25T22:58:17.190 INFO:teuthology.orchestra.run.smithi129.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-03-25T22:58:17.419 INFO:teuthology.orchestra.run.smithi129.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-03-25T22:58:17.858 INFO:teuthology.orchestra.run.smithi129.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-03-25T22:58:17.864 INFO:teuthology.orchestra.run.smithi129.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-03-25T22:58:18.155 INFO:teuthology.orchestra.run.smithi129.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-03-25T22:58:18.155 INFO:teuthology.orchestra.run.smithi129.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-03-25T22:58:18.157 INFO:teuthology.orchestra.run.smithi129.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-03-25T22:58:18.159 INFO:teuthology.orchestra.run.smithi129.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-03-25T22:58:18.160 INFO:teuthology.orchestra.run.smithi129.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-03-25T22:58:18.163 INFO:teuthology.orchestra.run.smithi129.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-03-25T22:58:18.164 INFO:teuthology.orchestra.run.smithi129.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-03-25T22:58:18.165 INFO:teuthology.orchestra.run.smithi129.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-03-25T22:58:18.347 INFO:teuthology.orchestra.run.smithi129.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-03-25T22:58:18.759 INFO:teuthology.orchestra.run.smithi129.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-03-25T22:58:18.859 INFO:teuthology.orchestra.run.smithi129.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-03-25T22:58:18.859 INFO:teuthology.orchestra.run.smithi129.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-03-25T22:58:18.991 INFO:teuthology.orchestra.run.smithi129.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-03-25T22:58:18.993 INFO:teuthology.orchestra.run.smithi129.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-03-25-0744/4354a20109fd25703f6611a1a8f196608a256149/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2416-g4354a201-1jammy [121 kB] 2024-03-25T22:58:18.995 INFO:teuthology.orchestra.run.smithi129.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-03-25-0744/4354a20109fd25703f6611a1a8f196608a256149/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2416-g4354a201-1jammy [38.9 kB] 2024-03-25T22:58:18.996 INFO:teuthology.orchestra.run.smithi129.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-03-25-0744/4354a20109fd25703f6611a1a8f196608a256149/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2416-g4354a201-1jammy [13.2 MB] 2024-03-25T22:58:18.997 INFO:teuthology.orchestra.run.smithi129.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-03-25T22:58:18.999 INFO:teuthology.orchestra.run.smithi129.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-03-25T22:58:19.003 INFO:teuthology.orchestra.run.smithi129.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-03-25T22:58:19.051 INFO:teuthology.orchestra.run.smithi129.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-03-25T22:58:19.054 INFO:teuthology.orchestra.run.smithi129.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-03-25T22:58:19.076 INFO:teuthology.orchestra.run.smithi129.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-03-25T22:58:19.239 INFO:teuthology.orchestra.run.smithi129.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-03-25T22:58:19.388 INFO:teuthology.orchestra.run.smithi129.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-03-25-0744/4354a20109fd25703f6611a1a8f196608a256149/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2416-g4354a201-1jammy [91.2 kB] 2024-03-25T22:58:19.512 INFO:teuthology.orchestra.run.smithi129.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-03-25T22:58:19.530 INFO:teuthology.orchestra.run.smithi129.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-03-25T22:58:19.683 INFO:teuthology.orchestra.run.smithi129.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-03-25T22:58:19.712 INFO:teuthology.orchestra.run.smithi129.stdout:Get:113 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-03-25T22:58:19.869 INFO:teuthology.orchestra.run.smithi129.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-03-25T22:58:19.872 INFO:teuthology.orchestra.run.smithi129.stdout:Get:115 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-03-25T22:58:19.874 INFO:teuthology.orchestra.run.smithi129.stdout:Get:116 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-03-25T22:58:19.896 INFO:teuthology.orchestra.run.smithi129.stdout:Get:117 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-03-25T22:58:19.901 INFO:teuthology.orchestra.run.smithi129.stdout:Get:118 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-03-25T22:58:20.547 INFO:teuthology.orchestra.run.smithi129.stdout:Fetched 178 MB in 7s (25.4 MB/s) 2024-03-25T22:58:20.707 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-03-25T22:58:20.808 INFO:teuthology.orchestra.run.smithi129.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165997 files and directories currently installed.) 2024-03-25T22:58:20.813 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-03-25T22:58:20.899 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-25T22:58:21.217 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-03-25T22:58:21.244 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-03-25T22:58:21.284 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-25T22:58:21.629 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-03-25T22:58:21.655 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-03-25T22:58:21.695 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-25T22:58:22.140 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-03-25T22:58:22.167 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-25T22:58:22.215 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-25T22:58:22.602 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-03-25T22:58:22.628 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-25T22:58:22.669 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-25T22:58:22.964 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-03-25T22:58:22.990 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-25T22:58:23.030 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-25T22:58:23.375 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-03-25T22:58:23.401 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-03-25T22:58:23.450 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-25T22:58:23.786 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package librados2. 2024-03-25T22:58:23.813 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../007-librados2_19.0.0-2416-g4354a201-1jammy_amd64.deb ... 2024-03-25T22:58:23.853 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking librados2 (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:58:24.256 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package librbd1. 2024-03-25T22:58:24.271 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../008-librbd1_19.0.0-2416-g4354a201-1jammy_amd64.deb ... 2024-03-25T22:58:24.305 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking librbd1 (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:58:24.750 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libcephfs2. 2024-03-25T22:58:24.777 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2416-g4354a201-1jammy_amd64.deb ... 2024-03-25T22:58:24.817 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libcephfs2 (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:58:25.103 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-rados. 2024-03-25T22:58:25.118 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2416-g4354a201-1jammy_amd64.deb ... 2024-03-25T22:58:25.161 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-rados (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:58:25.489 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-03-25T22:58:25.515 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2416-g4354a201-1jammy_all.deb ... 2024-03-25T22:58:25.555 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-ceph-argparse (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:58:25.766 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-cephfs. 2024-03-25T22:58:25.782 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2416-g4354a201-1jammy_amd64.deb ... 2024-03-25T22:58:25.825 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-cephfs (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:58:26.120 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-ceph-common. 2024-03-25T22:58:26.146 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2416-g4354a201-1jammy_all.deb ... 2024-03-25T22:58:26.186 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-ceph-common (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:58:26.689 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-wcwidth. 2024-03-25T22:58:26.716 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-03-25T22:58:26.747 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-25T22:58:27.034 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-prettytable. 2024-03-25T22:58:27.060 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-03-25T22:58:27.100 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-03-25T22:58:27.364 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-rbd. 2024-03-25T22:58:27.388 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2416-g4354a201-1jammy_amd64.deb ... 2024-03-25T22:58:27.428 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-rbd (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:58:27.773 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-03-25T22:58:27.799 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-03-25T22:58:27.839 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-25T22:58:28.134 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-03-25T22:58:28.160 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-03-25T22:58:28.200 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-03-25T22:58:28.503 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-03-25T22:58:28.530 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-03-25T22:58:28.570 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-25T22:58:28.898 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package lua5.1. 2024-03-25T22:58:28.913 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-03-25T22:58:28.956 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-03-25T22:58:29.268 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package lua-any. 2024-03-25T22:58:29.295 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-03-25T22:58:29.326 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking lua-any (27ubuntu1) ... 2024-03-25T22:58:29.554 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package zip. 2024-03-25T22:58:29.569 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-03-25T22:58:29.612 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking zip (3.0-12build2) ... 2024-03-25T22:58:29.924 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package unzip. 2024-03-25T22:58:29.950 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-03-25T22:58:29.990 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-03-25T22:58:30.370 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package luarocks. 2024-03-25T22:58:30.396 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-03-25T22:58:30.436 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-03-25T22:58:30.769 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package librgw2. 2024-03-25T22:58:30.785 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../025-librgw2_19.0.0-2416-g4354a201-1jammy_amd64.deb ... 2024-03-25T22:58:30.819 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking librgw2 (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:58:31.313 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-rgw. 2024-03-25T22:58:31.329 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2416-g4354a201-1jammy_amd64.deb ... 2024-03-25T22:58:31.372 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-rgw (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:58:31.708 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package liboath0:amd64. 2024-03-25T22:58:31.734 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-03-25T22:58:31.774 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-03-25T22:58:32.103 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libradosstriper1. 2024-03-25T22:58:32.130 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2416-g4354a201-1jammy_amd64.deb ... 2024-03-25T22:58:32.169 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libradosstriper1 (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:58:32.590 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-common. 2024-03-25T22:58:32.617 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2416-g4354a201-1jammy_amd64.deb ... 2024-03-25T22:58:32.656 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-common (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:58:33.887 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-base. 2024-03-25T22:58:33.914 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2416-g4354a201-1jammy_amd64.deb ... 2024-03-25T22:58:33.954 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-base (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:58:34.466 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-03-25T22:58:34.493 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-03-25T22:58:34.532 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-03-25T22:58:34.836 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-cheroot. 2024-03-25T22:58:34.863 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-03-25T22:58:34.902 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-25T22:58:35.230 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-03-25T22:58:35.257 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-03-25T22:58:35.297 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-03-25T22:58:35.575 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-jaraco.text. 2024-03-25T22:58:35.602 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-03-25T22:58:35.641 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-03-25T22:58:35.911 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-03-25T22:58:35.938 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-03-25T22:58:35.978 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-03-25T22:58:36.281 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-tempora. 2024-03-25T22:58:36.308 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-03-25T22:58:36.347 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-03-25T22:58:36.650 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-portend. 2024-03-25T22:58:36.677 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-03-25T22:58:36.717 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-03-25T22:58:37.170 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-03-25T22:58:37.197 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-03-25T22:58:37.237 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-03-25T22:58:37.540 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-cherrypy3. 2024-03-25T22:58:37.566 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-03-25T22:58:37.606 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-03-25T22:58:37.977 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-natsort. 2024-03-25T22:58:38.003 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-03-25T22:58:38.043 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-03-25T22:58:38.355 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-logutils. 2024-03-25T22:58:38.381 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-03-25T22:58:38.421 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-03-25T22:58:38.707 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-markupsafe. 2024-03-25T22:58:38.734 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-03-25T22:58:38.774 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-03-25T22:58:39.044 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-mako. 2024-03-25T22:58:39.059 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-03-25T22:58:39.094 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-25T22:58:39.414 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-simplegeneric. 2024-03-25T22:58:39.440 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-03-25T22:58:39.480 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-03-25T22:58:39.766 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-singledispatch. 2024-03-25T22:58:39.793 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-03-25T22:58:39.833 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-03-25T22:58:40.137 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-webob. 2024-03-25T22:58:40.162 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-03-25T22:58:40.202 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-03-25T22:58:40.539 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-waitress. 2024-03-25T22:58:40.565 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-03-25T22:58:40.609 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-25T22:58:40.875 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-tempita. 2024-03-25T22:58:40.902 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-03-25T22:58:40.933 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-03-25T22:58:41.212 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-paste. 2024-03-25T22:58:41.238 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-03-25T22:58:41.278 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-03-25T22:58:41.590 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-03-25T22:58:41.616 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-03-25T22:58:41.656 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-03-25T22:58:41.934 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-pastedeploy. 2024-03-25T22:58:41.960 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-03-25T22:58:41.992 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-03-25T22:58:42.290 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-webtest. 2024-03-25T22:58:42.316 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-03-25T22:58:42.356 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-03-25T22:58:42.667 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-pecan. 2024-03-25T22:58:42.693 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-03-25T22:58:42.733 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-03-25T22:58:43.072 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-certifi. 2024-03-25T22:58:43.098 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-03-25T22:58:43.130 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-03-25T22:58:43.449 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-idna. 2024-03-25T22:58:43.476 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-03-25T22:58:43.516 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-idna (3.3-1) ... 2024-03-25T22:58:43.818 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-urllib3. 2024-03-25T22:58:44.433 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-03-25T22:58:44.433 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-25T22:58:44.433 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-requests. 2024-03-25T22:58:44.433 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-03-25T22:58:44.433 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-25T22:58:44.631 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-werkzeug. 2024-03-25T22:58:44.651 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-03-25T22:58:44.696 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-25T22:58:44.987 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-03-25T22:58:45.013 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2416-g4354a201-1jammy_all.deb ... 2024-03-25T22:58:45.054 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:58:45.424 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-03-25T22:58:45.451 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2416-g4354a201-1jammy_amd64.deb ... 2024-03-25T22:58:45.490 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:58:45.776 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-bcrypt. 2024-03-25T22:58:45.792 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-03-25T22:58:45.826 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-03-25T22:58:46.096 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-openssl. 2024-03-25T22:58:46.123 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-03-25T22:58:46.513 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-03-25T22:58:47.633 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-mgr. 2024-03-25T22:58:47.648 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2416-g4354a201-1jammy_amd64.deb ... 2024-03-25T22:58:47.691 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-mgr (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:58:48.069 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-mon. 2024-03-25T22:58:48.096 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2416-g4354a201-1jammy_amd64.deb ... 2024-03-25T22:58:48.135 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-mon (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:58:48.748 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-osd. 2024-03-25T22:58:48.774 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2416-g4354a201-1jammy_amd64.deb ... 2024-03-25T22:58:48.814 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-osd (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:58:49.859 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph. 2024-03-25T22:58:49.874 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../066-ceph_19.0.0-2416-g4354a201-1jammy_amd64.deb ... 2024-03-25T22:58:49.917 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:58:50.270 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-fuse. 2024-03-25T22:58:50.296 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2416-g4354a201-1jammy_amd64.deb ... 2024-03-25T22:58:50.336 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-fuse (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:58:50.673 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-mds. 2024-03-25T22:58:50.699 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2416-g4354a201-1jammy_amd64.deb ... 2024-03-25T22:58:50.731 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-mds (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:58:51.109 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package cephadm. 2024-03-25T22:58:51.135 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../069-cephadm_19.0.0-2416-g4354a201-1jammy_amd64.deb ... 2024-03-25T22:58:51.175 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking cephadm (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:58:51.486 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-asyncssh. 2024-03-25T22:58:51.513 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-03-25T22:58:51.553 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-03-25T22:58:51.855 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-babel-localedata. 2024-03-25T22:58:51.871 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-03-25T22:58:51.905 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-25T22:58:52.839 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-babel. 2024-03-25T22:58:52.862 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-03-25T22:58:52.902 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-03-25T22:58:53.230 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-jinja2. 2024-03-25T22:58:53.257 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-03-25T22:58:53.296 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-25T22:58:53.641 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-03-25T22:58:53.668 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2416-g4354a201-1jammy_all.deb ... 2024-03-25T22:58:53.708 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:58:54.011 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-repoze.lru. 2024-03-25T22:58:54.037 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-03-25T22:58:54.110 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-03-25T22:58:54.422 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-routes. 2024-03-25T22:58:54.449 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-03-25T22:58:54.488 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-03-25T22:58:54.825 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-03-25T22:58:54.852 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2416-g4354a201-1jammy_all.deb ... 2024-03-25T22:58:54.891 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:58:56.114 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-03-25T22:58:56.140 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-03-25T22:58:56.189 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-25T22:58:56.675 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-joblib. 2024-03-25T22:58:56.702 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-03-25T22:58:56.741 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-03-25T22:58:57.111 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-03-25T22:58:57.138 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-03-25T22:58:57.177 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-03-25T22:58:57.489 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-sklearn. 2024-03-25T22:58:57.516 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-03-25T22:58:57.555 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-25T22:58:58.564 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-03-25T22:58:58.591 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2416-g4354a201-1jammy_all.deb ... 2024-03-25T22:58:58.715 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:58:59.661 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-cachetools. 2024-03-25T22:58:59.689 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-03-25T22:58:59.727 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-03-25T22:59:00.006 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-pyasn1. 2024-03-25T22:59:00.033 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-03-25T22:59:00.063 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-03-25T22:59:00.384 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-03-25T22:59:00.411 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-03-25T22:59:00.450 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-03-25T22:59:00.761 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-rsa. 2024-03-25T22:59:00.789 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-03-25T22:59:00.827 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-rsa (4.8-1) ... 2024-03-25T22:59:01.157 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-google-auth. 2024-03-25T22:59:01.184 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-03-25T22:59:01.223 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-03-25T22:59:01.551 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-03-25T22:59:01.579 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-03-25T22:59:01.617 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-25T22:59:01.887 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-websocket. 2024-03-25T22:59:01.915 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-03-25T22:59:01.945 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-03-25T22:59:02.232 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-kubernetes. 2024-03-25T22:59:02.260 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-03-25T22:59:02.307 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-25T22:59:03.004 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-03-25T22:59:03.032 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2416-g4354a201-1jammy_all.deb ... 2024-03-25T22:59:03.070 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:59:03.399 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libonig5:amd64. 2024-03-25T22:59:03.426 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-03-25T22:59:03.465 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-25T22:59:03.793 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libjq1:amd64. 2024-03-25T22:59:03.821 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-03-25T22:59:03.859 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-25T22:59:04.087 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package jq. 2024-03-25T22:59:04.103 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-03-25T22:59:04.145 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-03-25T22:59:04.398 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package socat. 2024-03-25T22:59:04.414 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-03-25T22:59:04.457 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-03-25T22:59:04.768 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package xmlstarlet. 2024-03-25T22:59:04.795 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-03-25T22:59:04.834 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-03-25T22:59:05.172 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-test. 2024-03-25T22:59:05.199 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2416-g4354a201-1jammy_amd64.deb ... 2024-03-25T22:59:05.238 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-test (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:59:07.576 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-volume. 2024-03-25T22:59:07.603 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2416-g4354a201-1jammy_all.deb ... 2024-03-25T22:59:07.736 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-volume (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:59:08.054 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libcephfs-dev. 2024-03-25T22:59:08.070 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2416-g4354a201-1jammy_amd64.deb ... 2024-03-25T22:59:08.112 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libcephfs-dev (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:59:08.516 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package lua-socket:amd64. 2024-03-25T22:59:08.543 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-03-25T22:59:08.682 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-25T22:59:09.003 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package lua-sec:amd64. 2024-03-25T22:59:09.030 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-03-25T22:59:09.070 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-03-25T22:59:09.415 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package nvme-cli. 2024-03-25T22:59:09.443 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-03-25T22:59:09.481 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-03-25T22:59:09.853 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package pkg-config. 2024-03-25T22:59:09.869 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-03-25T22:59:09.903 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-03-25T22:59:10.139 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-03-25T22:59:10.155 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-03-25T22:59:10.197 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-03-25T22:59:10.617 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-iniconfig. 2024-03-25T22:59:10.645 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-03-25T22:59:10.683 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-03-25T22:59:10.986 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-pastescript. 2024-03-25T22:59:11.014 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-03-25T22:59:11.053 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-03-25T22:59:11.381 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-pluggy. 2024-03-25T22:59:11.409 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-03-25T22:59:11.448 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-03-25T22:59:11.751 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-psutil. 2024-03-25T22:59:11.778 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-03-25T22:59:11.817 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-03-25T22:59:12.162 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-py. 2024-03-25T22:59:12.189 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-03-25T22:59:12.228 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-py (1.10.0-1) ... 2024-03-25T22:59:12.565 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-pygments. 2024-03-25T22:59:12.592 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-03-25T22:59:12.648 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-03-25T22:59:13.085 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-pyinotify. 2024-03-25T22:59:13.113 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-03-25T22:59:13.143 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-03-25T22:59:13.446 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-toml. 2024-03-25T22:59:13.473 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-03-25T22:59:13.512 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-03-25T22:59:13.815 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-pytest. 2024-03-25T22:59:13.843 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-03-25T22:59:13.882 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-03-25T22:59:14.202 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-simplejson. 2024-03-25T22:59:14.229 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-03-25T22:59:14.268 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-03-25T22:59:14.520 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-03-25T22:59:14.537 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-03-25T22:59:14.604 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-03-25T22:59:15.266 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package radosgw. 2024-03-25T22:59:15.292 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../116-radosgw_19.0.0-2416-g4354a201-1jammy_amd64.deb ... 2024-03-25T22:59:15.333 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking radosgw (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:59:16.102 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package rbd-fuse. 2024-03-25T22:59:16.118 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2416-g4354a201-1jammy_amd64.deb ... 2024-03-25T22:59:16.160 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking rbd-fuse (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:59:16.457 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-03-25T22:59:16.715 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-25T22:59:16.841 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-03-25T22:59:17.086 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-03-25T22:59:17.438 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-03-25T22:59:17.866 INFO:teuthology.orchestra.run.smithi129.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-03-25T22:59:17.952 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up cephadm (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:59:18.769 INFO:teuthology.orchestra.run.smithi129.stdout:Adding system user cephadm....done 2024-03-25T22:59:18.889 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-25T22:59:19.402 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-03-25T22:59:19.644 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-03-25T22:59:19.745 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-03-25T22:59:19.980 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-03-25T22:59:20.231 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-03-25T22:59:20.357 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-py (1.10.0-1) ... 2024-03-25T22:59:20.692 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-03-25T22:59:21.044 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-03-25T22:59:21.287 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-03-25T22:59:21.554 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-03-25T22:59:21.806 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-03-25T22:59:22.057 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-03-25T22:59:22.359 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-03-25T22:59:22.609 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-ceph-argparse (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:59:22.860 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-25T22:59:22.961 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-25T22:59:23.062 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-03-25T22:59:23.171 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-03-25T22:59:23.431 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-03-25T22:59:23.805 INFO:teuthology.orchestra.run.smithi129.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-03-25T22:59:23.837 INFO:teuthology.orchestra.run.smithi129.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-03-25T22:59:23.884 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-25T22:59:24.010 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-03-25T22:59:24.370 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-25T22:59:24.470 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-03-25T22:59:24.730 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-03-25T22:59:25.000 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-03-25T22:59:25.301 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up zip (3.0-12build2) ... 2024-03-25T22:59:25.427 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-03-25T22:59:26.062 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-03-25T22:59:26.338 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-03-25T22:59:26.464 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-03-25T22:59:26.590 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-25T22:59:26.883 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-03-25T22:59:27.259 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-03-25T22:59:27.510 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-03-25T22:59:27.903 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-25T22:59:28.171 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-25T22:59:28.514 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-03-25T22:59:28.773 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-03-25T22:59:28.874 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-ceph-common (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:59:29.159 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-idna (3.3-1) ... 2024-03-25T22:59:29.468 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-25T22:59:30.632 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-03-25T22:59:30.917 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-25T22:59:31.068 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-toml (0.10.2-1) ... 2024-03-25T22:59:31.327 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-25T22:59:31.453 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-03-25T22:59:31.579 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-25T22:59:31.880 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-03-25T22:59:32.131 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-03-25T22:59:32.366 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-25T22:59:32.467 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-03-25T22:59:32.760 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-03-25T22:59:33.011 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-03-25T22:59:33.262 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-03-25T22:59:33.513 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-03-25T22:59:33.764 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-03-25T22:59:34.024 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-25T22:59:34.133 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-03-25T22:59:34.392 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-25T22:59:34.527 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-25T22:59:34.819 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-03-25T22:59:35.104 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-03-25T22:59:35.355 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-25T22:59:35.481 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-03-25T22:59:35.607 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-03-25T22:59:35.697 INFO:teuthology.orchestra.run.smithi129.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-03-25T22:59:35.942 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-25T22:59:36.060 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-03-25T22:59:36.461 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-03-25T22:59:36.720 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up lua-any (27ubuntu1) ... 2024-03-25T22:59:36.846 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-portend (3.0.0-1) ... 2024-03-25T22:59:37.099 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-25T22:59:37.223 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-25T22:59:37.549 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-03-25T22:59:37.867 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-25T22:59:38.168 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-03-25T22:59:38.294 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-rsa (4.8-1) ... 2024-03-25T22:59:38.562 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-03-25T22:59:38.813 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-25T22:59:39.182 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-03-25T22:59:39.570 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-03-25T22:59:39.869 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-03-25T22:59:40.221 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-25T22:59:40.347 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up librados2 (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:59:40.474 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:59:40.601 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-03-25T22:59:40.937 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libcephfs2 (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:59:41.045 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libradosstriper1 (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:59:41.146 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-03-25T22:59:41.424 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up librbd1 (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:59:41.549 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-mgr-modules-core (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:59:41.675 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-fuse (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:59:41.890 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-25T22:59:41.891 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-25T22:59:42.359 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libcephfs-dev (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:59:42.486 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-rados (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:59:42.595 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up librgw2 (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:59:42.720 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-25T22:59:43.932 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-rbd (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:59:44.042 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up rbd-fuse (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:59:44.159 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-rgw (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:59:44.285 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-cephfs (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:59:44.411 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-common (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:59:44.962 INFO:teuthology.orchestra.run.smithi129.stdout:Adding group ceph....done 2024-03-25T22:59:45.331 INFO:teuthology.orchestra.run.smithi129.stdout:Adding system user ceph....done 2024-03-25T22:59:45.449 INFO:teuthology.orchestra.run.smithi129.stdout:Setting system user ceph properties....done 2024-03-25T22:59:45.468 INFO:teuthology.orchestra.run.smithi129.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-03-25T22:59:45.602 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-03-25T22:59:45.931 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-03-25T22:59:46.404 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-test (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:59:46.530 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up radosgw (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:59:47.007 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-25T22:59:47.007 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-25T22:59:47.481 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-base (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:59:47.823 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-03-25T22:59:48.283 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-mds (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:59:48.519 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-25T22:59:48.520 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-25T22:59:49.010 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-mgr (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:59:49.230 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-25T22:59:49.230 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-25T22:59:49.711 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-osd (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:59:49.983 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-25T22:59:49.983 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-25T22:59:50.463 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:59:50.590 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:59:50.748 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-mon (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:59:50.968 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-25T22:59:50.968 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-25T22:59:51.505 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-mgr-cephadm (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:59:51.755 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:59:51.982 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-mgr-dashboard (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:59:52.223 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-volume (19.0.0-2416-g4354a201-1jammy) ... 2024-03-25T22:59:52.623 INFO:teuthology.orchestra.run.smithi129.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-03-25T22:59:52.732 INFO:teuthology.orchestra.run.smithi129.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-25T22:59:52.884 INFO:teuthology.orchestra.run.smithi129.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-25T22:59:53.659 INFO:teuthology.orchestra.run.smithi129.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-25T22:59:54.301 INFO:teuthology.orchestra.run.smithi129.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-25T22:59:54.306 DEBUG:teuthology.parallel:result is None 2024-03-25T22:59:54.307 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=4354a20109fd25703f6611a1a8f196608a256149 2024-03-25T22:59:54.527 DEBUG:teuthology.orchestra.run.smithi129:> dpkg-query -W -f '${Version}' ceph 2024-03-25T22:59:54.547 INFO:teuthology.orchestra.run.smithi129.stdout:19.0.0-2416-g4354a201-1jammy 2024-03-25T22:59:54.548 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2416-g4354a201-1jammy 2024-03-25T22:59:54.548 INFO:teuthology.task.install:The correct ceph version 19.0.0-2416-g4354a201-1jammy is installed. 2024-03-25T22:59:54.550 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T22:59:54.550 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-25T22:59:54.550 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T22:59:54.601 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T22:59:54.601 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-25T22:59:54.601 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T22:59:54.655 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T22:59:54.709 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T22:59:54.709 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-25T22:59:54.709 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T22:59:54.763 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T22:59:54.816 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T22:59:54.816 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-25T22:59:54.816 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T22:59:54.872 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T22:59:54.928 INFO:teuthology.run_tasks:Running task workunit... 2024-03-25T22:59:54.942 INFO:tasks.workunit:Pulling workunits from ref 4354a20109fd25703f6611a1a8f196608a256149 2024-03-25T22:59:54.943 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-25T22:59:54.943 INFO:tasks.workunit:timeout=3h 2024-03-25T22:59:54.943 INFO:tasks.workunit:cleanup=True 2024-03-25T22:59:54.943 DEBUG:teuthology.orchestra.run.smithi129:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-25T22:59:54.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T22:59:54.977 INFO:teuthology.orchestra.run.smithi129.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-25T22:59:54.977 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-03-25T22:59:55.024 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-03-25T22:59:55.025 DEBUG:teuthology.orchestra.run.smithi129:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-03-25T22:59:55.073 DEBUG:teuthology.orchestra.run.smithi129:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 4354a20109fd25703f6611a1a8f196608a256149 2024-03-25T22:59:55.123 INFO:tasks.workunit.client.0.smithi129.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-25T23:00:37.560 INFO:tasks.workunit.client.0.smithi129.stderr:Note: switching to '4354a20109fd25703f6611a1a8f196608a256149'. 2024-03-25T23:00:37.561 INFO:tasks.workunit.client.0.smithi129.stderr: 2024-03-25T23:00:37.561 INFO:tasks.workunit.client.0.smithi129.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-25T23:00:37.561 INFO:tasks.workunit.client.0.smithi129.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-25T23:00:37.561 INFO:tasks.workunit.client.0.smithi129.stderr:state without impacting any branches by switching back to a branch. 2024-03-25T23:00:37.561 INFO:tasks.workunit.client.0.smithi129.stderr: 2024-03-25T23:00:37.561 INFO:tasks.workunit.client.0.smithi129.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-25T23:00:37.561 INFO:tasks.workunit.client.0.smithi129.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-25T23:00:37.561 INFO:tasks.workunit.client.0.smithi129.stderr: 2024-03-25T23:00:37.561 INFO:tasks.workunit.client.0.smithi129.stderr: git switch -c 2024-03-25T23:00:37.561 INFO:tasks.workunit.client.0.smithi129.stderr: 2024-03-25T23:00:37.561 INFO:tasks.workunit.client.0.smithi129.stderr:Or undo this operation with: 2024-03-25T23:00:37.561 INFO:tasks.workunit.client.0.smithi129.stderr: 2024-03-25T23:00:37.561 INFO:tasks.workunit.client.0.smithi129.stderr: git switch - 2024-03-25T23:00:37.562 INFO:tasks.workunit.client.0.smithi129.stderr: 2024-03-25T23:00:37.562 INFO:tasks.workunit.client.0.smithi129.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-25T23:00:37.562 INFO:tasks.workunit.client.0.smithi129.stderr: 2024-03-25T23:00:37.562 INFO:tasks.workunit.client.0.smithi129.stderr:HEAD is now at 4354a20109f Merge branch 'wip-nitzan-init-hash-in-chunk-ref-by-hash' of https://github.com/NitzanMordhai/ceph into wip-yuri3-testing-2024-03-25-0744 2024-03-25T23:00:37.567 DEBUG:teuthology.orchestra.run.smithi129:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-25T23:00:37.615 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-25T23:00:37.615 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-25T23:00:37.661 INFO:tasks.workunit:Running workunits matching scrub on client.0... 2024-03-25T23:00:37.661 INFO:tasks.workunit:Running workunit scrub/osd-mapper.sh... 2024-03-25T23:00:37.661 DEBUG:teuthology.orchestra.run.smithi129:workunit test scrub/osd-mapper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4354a20109fd25703f6611a1a8f196608a256149 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh 2024-03-25T23:00:37.721 INFO:tasks.workunit.client.0.smithi129.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-03-25T23:00:37.726 INFO:tasks.workunit.client.0.smithi129.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-03-25T23:00:37.726 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-03-25T23:00:37.726 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-03-25T23:00:37.726 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-03-25T23:00:37.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-03-25T23:00:37.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-03-25T23:00:37.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-03-25T23:00:37.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-03-25T23:00:37.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-03-25T23:00:37.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-mapper 2024-03-25T23:00:37.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:11: run: local dir=td/osd-mapper 2024-03-25T23:00:37.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:12: run: shift 2024-03-25T23:00:37.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: export CEPH_MON=127.0.0.1:7144 2024-03-25T23:00:37.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: CEPH_MON=127.0.0.1:7144 2024-03-25T23:00:37.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:15: run: export CEPH_ARGS 2024-03-25T23:00:37.727 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: uuidgen 2024-03-25T23:00:37.728 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: CEPH_ARGS+='--fsid=99b4bc82-c34e-4895-95ea-2717aa85e928 --auth-supported=none ' 2024-03-25T23:00:37.728 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:17: run: CEPH_ARGS+='--mon-host=127.0.0.1:7144 ' 2024-03-25T23:00:37.728 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:19: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-03-25T23:00:37.728 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: set 2024-03-25T23:00:37.728 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-03-25T23:00:37.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: local funcs=TEST_truncated_sna_record 2024-03-25T23:00:37.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:21: run: for func in $funcs 2024-03-25T23:00:37.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:22: run: setup td/osd-mapper 2024-03-25T23:00:37.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-mapper 2024-03-25T23:00:37.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-mapper 2024-03-25T23:00:37.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-03-25T23:00:37.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T23:00:37.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-03-25T23:00:37.731 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:00:37.732 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:00:37.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:00:37.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:00:37.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:00:37.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:00:37.732 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T23:00:37.733 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T23:00:37.733 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T23:00:37.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T23:00:37.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T23:00:37.734 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T23:00:37.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:00:37.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T23:00:37.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T23:00:37.736 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:00:37.737 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T23:00:37.739 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T23:00:37.739 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-03-25T23:00:37.739 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T23:00:37.739 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:00:37.739 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20201 2024-03-25T23:00:37.739 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20201 2024-03-25T23:00:37.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T23:00:37.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T23:00:37.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-mapper 2024-03-25T23:00:37.740 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-25T23:00:37.741 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:00:37.741 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20201 2024-03-25T23:00:37.741 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20201 2024-03-25T23:00:37.743 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-25T23:00:37.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-03-25T23:00:37.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-03-25T23:00:37.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-25T23:00:37.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-mapper 1' TERM HUP INT 2024-03-25T23:00:37.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:23: run: TEST_truncated_sna_record td/osd-mapper 2024-03-25T23:00:37.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:45: TEST_truncated_sna_record: local dir=td/osd-mapper 2024-03-25T23:00:37.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='4' ['pool_name']='test') 2024-03-25T23:00:37.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: local -A cluster_conf 2024-03-25T23:00:37.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:52: TEST_truncated_sna_record: local extr_dbg=3 2024-03-25T23:00:37.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: (( extr_dbg > 1 )) 2024-03-25T23:00:37.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: echo 'Dir: td/osd-mapper' 2024-03-25T23:00:37.744 INFO:tasks.workunit.client.0.smithi129.stdout:Dir: td/osd-mapper 2024-03-25T23:00:37.745 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:54: TEST_truncated_sna_record: standard_scrub_cluster td/osd-mapper cluster_conf 2024-03-25T23:00:37.745 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:221: standard_scrub_cluster: local dir=td/osd-mapper 2024-03-25T23:00:37.745 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:222: standard_scrub_cluster: local -n args=cluster_conf 2024-03-25T23:00:37.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:224: standard_scrub_cluster: local OSDS=3 2024-03-25T23:00:37.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:225: standard_scrub_cluster: local pg_num=4 2024-03-25T23:00:37.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:226: standard_scrub_cluster: local poolname=test 2024-03-25T23:00:37.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:227: standard_scrub_cluster: args['pool_name']=test 2024-03-25T23:00:37.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:228: standard_scrub_cluster: local extra_pars= 2024-03-25T23:00:37.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local debug_msg=dbg 2024-03-25T23:00:37.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local saved_echo_flag=x 2024-03-25T23:00:37.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: set +x 2024-03-25T23:00:39.671 INFO:tasks.workunit.client.0.smithi129.stdout:add osd0 d0f8b0f8-175f-43ff-920c-011f8bc0c712 2024-03-25T23:00:39.921 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:00:40.019 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:00:40.017+0000 7f063a7857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:00:40.040 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:00:40.041+0000 7f063a7857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:00:40.042 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:00:40.041+0000 7f063a7857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:00:40.042 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:00:40.041+0000 7f063a7857c0 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-03-25T23:00:40.042 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:00:40.041+0000 7f063a7857c0 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-03-25T23:00:40.042 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:00:40.041+0000 7f063a7857c0 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-03-25T23:00:40.072 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:00:40.073+0000 7f063a7857c0 -1 bluestore(td/osd-mapper/0) _read_fsid unparsable uuid 2024-03-25T23:00:46.075 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd0 key to auth repository 2024-03-25T23:00:46.276 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.0 2024-03-25T23:00:46.310 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:00:46.309+0000 7f33eac7a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:00:46.344 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:00:46.345+0000 7f33eac7a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:00:46.345 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:00:46.345+0000 7f33eac7a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:00:46.492 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:00:46.941 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:00:46.941+0000 7f33eac7a7c0 -1 Falling back to public interface 2024-03-25T23:00:47.708 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:00:48.934 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:00:49.614 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:00:49.613+0000 7f33eac7a7c0 -1 osd.0 0 log_to_monitors true 2024-03-25T23:00:50.234 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:00:51.611 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:00:52.932 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:00:54.351 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:00:55.658 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:00:56.966 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:00:58.401 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:00:59.712 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:01:01.016 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:01:02.645 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:01:04.234 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:01:05.618 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:01:06.935 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:01:08.452 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:01:09.972 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:01:11.919 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:01:14.178 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:01:14.487 INFO:tasks.workunit.client.0.smithi129.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1514323707,v1:127.0.0.1:6801/1514323707] [v2:127.0.0.1:6802/1514323707,v1:127.0.0.1:6803/1514323707] exists,up d0f8b0f8-175f-43ff-920c-011f8bc0c712 2024-03-25T23:01:14.491 INFO:tasks.workunit.client.0.smithi129.stdout:add osd1 7ef91d1c-2bf1-4f0c-86db-44d843c5755b 2024-03-25T23:01:14.936 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:01:14.997 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:01:14.997+0000 7f1fe31a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:01:15.003 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:01:15.005+0000 7f1fe31a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:01:15.005 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:01:15.005+0000 7f1fe31a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:01:15.005 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:01:15.005+0000 7f1fe31a87c0 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-03-25T23:01:15.005 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:01:15.005+0000 7f1fe31a87c0 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-03-25T23:01:15.005 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:01:15.005+0000 7f1fe31a87c0 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-03-25T23:01:15.073 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:01:15.073+0000 7f1fe31a87c0 -1 bluestore(td/osd-mapper/1) _read_fsid unparsable uuid 2024-03-25T23:01:20.593 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd1 key to auth repository 2024-03-25T23:01:20.987 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.1 2024-03-25T23:01:21.021 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:01:21.021+0000 7f55aab687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:01:21.050 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:01:21.049+0000 7f55aab687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:01:21.051 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:01:21.053+0000 7f55aab687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:01:21.313 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:01:21.597 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:01:21.597+0000 7f55aab687c0 -1 Falling back to public interface 2024-03-25T23:01:22.637 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:01:23.952 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:01:24.160 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:01:24.161+0000 7f55aab687c0 -1 osd.1 0 log_to_monitors true 2024-03-25T23:01:25.273 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:01:26.592 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:01:28.008 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:01:29.322 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:01:30.643 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:01:31.986 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:01:33.306 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:01:34.621 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:01:36.012 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:01:37.333 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:01:38.999 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:01:40.483 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:01:42.308 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:01:44.250 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:01:45.785 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:01:47.498 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:01:48.894 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:01:49.211 INFO:tasks.workunit.client.0.smithi129.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4077916633,v1:127.0.0.1:6811/4077916633] [v2:127.0.0.1:6812/4077916633,v1:127.0.0.1:6813/4077916633] exists,up 7ef91d1c-2bf1-4f0c-86db-44d843c5755b 2024-03-25T23:01:49.218 INFO:tasks.workunit.client.0.smithi129.stdout:add osd2 6a2a0a84-9b88-4dbf-ae30-3c6f603fd676 2024-03-25T23:01:49.584 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:01:49.641 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:01:49.639+0000 7efc41bf27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:01:49.652 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:01:49.647+0000 7efc41bf27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:01:49.653 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:01:49.651+0000 7efc41bf27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:01:49.653 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:01:49.651+0000 7efc41bf27c0 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-03-25T23:01:49.654 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:01:49.651+0000 7efc41bf27c0 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-03-25T23:01:49.654 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:01:49.651+0000 7efc41bf27c0 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-03-25T23:01:49.722 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:01:49.719+0000 7efc41bf27c0 -1 bluestore(td/osd-mapper/2) _read_fsid unparsable uuid 2024-03-25T23:01:56.418 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd2 key to auth repository 2024-03-25T23:01:56.834 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.2 2024-03-25T23:01:56.868 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:01:56.863+0000 7f2776ecc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:01:56.870 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:01:56.867+0000 7f2776ecc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:01:56.871 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:01:56.867+0000 7f2776ecc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:01:57.144 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:01:57.407 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:01:57.403+0000 7f2776ecc7c0 -1 Falling back to public interface 2024-03-25T23:01:58.446 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:01:59.778 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:02:00.255 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:00.251+0000 7f2776ecc7c0 -1 osd.2 0 log_to_monitors true 2024-03-25T23:02:01.096 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:02:02.573 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:02:03.892 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:02:05.421 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:02:06.730 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:02:08.126 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:02:09.431 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:02:10.748 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:02:12.069 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:02:13.399 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:02:14.712 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:02:16.575 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:02:18.028 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:02:19.445 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:02:21.005 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:02:22.435 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:02:24.089 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:02:25.857 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-25T23:02:27.319 INFO:tasks.workunit.client.0.smithi129.stdout:21 2024-03-25T23:02:27.665 INFO:tasks.workunit.client.0.smithi129.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2728860577,v1:127.0.0.1:6819/2728860577] [v2:127.0.0.1:6820/2728860577,v1:127.0.0.1:6821/2728860577] exists,up 6a2a0a84-9b88-4dbf-ae30-3c6f603fd676 2024-03-25T23:02:28.079 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'test' created 2024-03-25T23:02:30.228 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 21474836503 2024-03-25T23:02:30.554 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 42949672978 2024-03-25T23:02:32.221 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 64424509448 2024-03-25T23:02:34.894 INFO:tasks.workunit.client.0.smithi129.stdout:standard_scrub_cluster: dbg: test pool is test 1 2024-03-25T23:02:34.895 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:55: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_not_scrubbing 1 2024-03-25T23:02:35.029 INFO:tasks.workunit.client.0.smithi129.stdout:osd.0: { 2024-03-25T23:02:35.030 INFO:tasks.workunit.client.0.smithi129.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-03-25T23:02:35.030 INFO:tasks.workunit.client.0.smithi129.stdout:} 2024-03-25T23:02:35.043 INFO:tasks.workunit.client.0.smithi129.stdout:osd.1: { 2024-03-25T23:02:35.043 INFO:tasks.workunit.client.0.smithi129.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-03-25T23:02:35.043 INFO:tasks.workunit.client.0.smithi129.stdout:} 2024-03-25T23:02:35.056 INFO:tasks.workunit.client.0.smithi129.stdout:osd.2: { 2024-03-25T23:02:35.056 INFO:tasks.workunit.client.0.smithi129.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-03-25T23:02:35.056 INFO:tasks.workunit.client.0.smithi129.stdout:} 2024-03-25T23:02:35.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:56: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_scrubbing 1 2024-03-25T23:02:35.197 INFO:tasks.workunit.client.0.smithi129.stdout:osd.0: { 2024-03-25T23:02:35.198 INFO:tasks.workunit.client.0.smithi129.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-03-25T23:02:35.198 INFO:tasks.workunit.client.0.smithi129.stdout:} 2024-03-25T23:02:35.210 INFO:tasks.workunit.client.0.smithi129.stdout:osd.1: { 2024-03-25T23:02:35.210 INFO:tasks.workunit.client.0.smithi129.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-03-25T23:02:35.211 INFO:tasks.workunit.client.0.smithi129.stdout:} 2024-03-25T23:02:35.223 INFO:tasks.workunit.client.0.smithi129.stdout:osd.2: { 2024-03-25T23:02:35.223 INFO:tasks.workunit.client.0.smithi129.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-03-25T23:02:35.223 INFO:tasks.workunit.client.0.smithi129.stdout:} 2024-03-25T23:02:35.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:58: TEST_truncated_sna_record: local osdn=3 2024-03-25T23:02:35.239 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:59: TEST_truncated_sna_record: local poolid=1 2024-03-25T23:02:35.239 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:60: TEST_truncated_sna_record: local poolname=test 2024-03-25T23:02:35.239 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:61: TEST_truncated_sna_record: local objname=objxxx 2024-03-25T23:02:35.239 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:64: TEST_truncated_sna_record: make_a_clone test objxxx snap01 snap02 2024-03-25T23:02:35.239 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-03-25T23:02:35.239 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-03-25T23:02:35.565 INFO:tasks.workunit.client.0.smithi129.stdout:created pool test snap snap01 2024-03-25T23:02:35.787 INFO:tasks.workunit.client.0.smithi129.stdout:created pool test snap snap02 2024-03-25T23:02:35.791 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:65: TEST_truncated_sna_record: make_a_clone test objxxx snap13 2024-03-25T23:02:35.791 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-03-25T23:02:35.791 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-03-25T23:02:36.327 INFO:tasks.workunit.client.0.smithi129.stdout:created pool test snap snap13 2024-03-25T23:02:36.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:66: TEST_truncated_sna_record: make_a_clone test objxxx snap24 snap25 2024-03-25T23:02:36.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-03-25T23:02:36.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-03-25T23:02:36.913 INFO:tasks.workunit.client.0.smithi129.stdout:created pool test snap snap24 2024-03-25T23:02:37.119 INFO:tasks.workunit.client.0.smithi129.stdout:created pool test snap snap25 2024-03-25T23:02:37.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: echo 30969 2024-03-25T23:02:37.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: rados -p test put objxxx - 2024-03-25T23:02:37.463 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-03-25T23:02:37.463 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: jq -r .pgid 2024-03-25T23:02:37.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: local pgid=1.3 2024-03-25T23:02:37.785 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-03-25T23:02:37.785 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: jq -r '.up[0]' 2024-03-25T23:02:38.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: local osd=1 2024-03-25T23:02:38.135 INFO:tasks.workunit.client.0.smithi129.stdout:pgid is 1.3 (primary: osd.1) 2024-03-25T23:02:38.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:72: TEST_truncated_sna_record: echo 'pgid is 1.3 (primary: osd.1)' 2024-03-25T23:02:38.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:74: TEST_truncated_sna_record: set_query_debug 1.3 2024-03-25T23:02:38.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:295: set_query_debug: local pgid=1.3 2024-03-25T23:02:38.136 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:297: set_query_debug: awk -v 'pg=^1.3' -n -e '$0 ~ pg { print(gensub(/[^0-9]*([0-9]+).*/,"\\1","g",$5)); }' 2024-03-25T23:02:38.136 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:297: set_query_debug: ceph pg dump pgs_brief 2024-03-25T23:02:38.426 INFO:tasks.workunit.client.0.smithi129.stderr:dumped pgs_brief 2024-03-25T23:02:38.443 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:297: set_query_debug: local prim_osd=1 2024-03-25T23:02:38.443 INFO:tasks.workunit.client.0.smithi129.stdout:Setting scrub debug data. Primary for 1.3 is 1 2024-03-25T23:02:38.443 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:299: set_query_debug: echo 'Setting scrub debug data. Primary for 1.3 is 1' 2024-03-25T23:02:38.444 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:300: set_query_debug: get_asok_path osd.1 2024-03-25T23:02:38.444 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-25T23:02:38.444 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-25T23:02:38.445 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T23:02:38.445 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:02:38.445 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20201 2024-03-25T23:02:38.445 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20201/ceph-osd.1.asok 2024-03-25T23:02:38.446 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:300: set_query_debug: CEPH_ARGS= 2024-03-25T23:02:38.446 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:300: set_query_debug: ceph --format=json daemon /tmp/ceph-asok.20201/ceph-osd.1.asok scrubdebug 1.3 set sessions 2024-03-25T23:02:38.555 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-25T23:02:38.555 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2024-03-25T23:02:38.590 INFO:tasks.workunit.client.0.smithi129.stdout:{"success":true}{ 2024-03-25T23:02:38.590 INFO:tasks.workunit.client.0.smithi129.stdout: "name": "objxxx", 2024-03-25T23:02:38.590 INFO:tasks.workunit.client.0.smithi129.stdout: "clones": [ 2024-03-25T23:02:38.590 INFO:tasks.workunit.client.0.smithi129.stdout: { 2024-03-25T23:02:38.590 INFO:tasks.workunit.client.0.smithi129.stdout: "id": 2, 2024-03-25T23:02:38.590 INFO:tasks.workunit.client.0.smithi129.stdout: "snapshots": [ 2024-03-25T23:02:38.590 INFO:tasks.workunit.client.0.smithi129.stdout: { 2024-03-25T23:02:38.590 INFO:tasks.workunit.client.0.smithi129.stdout: "id": 1, 2024-03-25T23:02:38.590 INFO:tasks.workunit.client.0.smithi129.stdout: "name": "snap01" 2024-03-25T23:02:38.591 INFO:tasks.workunit.client.0.smithi129.stdout: }, 2024-03-25T23:02:38.591 INFO:tasks.workunit.client.0.smithi129.stdout: { 2024-03-25T23:02:38.591 INFO:tasks.workunit.client.0.smithi129.stdout: "id": 2, 2024-03-25T23:02:38.591 INFO:tasks.workunit.client.0.smithi129.stdout: "name": "snap02" 2024-03-25T23:02:38.591 INFO:tasks.workunit.client.0.smithi129.stdout: } 2024-03-25T23:02:38.591 INFO:tasks.workunit.client.0.smithi129.stdout: ], 2024-03-25T23:02:38.591 INFO:tasks.workunit.client.0.smithi129.stdout: "size": 6, 2024-03-25T23:02:38.591 INFO:tasks.workunit.client.0.smithi129.stdout: "overlaps": [] 2024-03-25T23:02:38.591 INFO:tasks.workunit.client.0.smithi129.stdout: }, 2024-03-25T23:02:38.591 INFO:tasks.workunit.client.0.smithi129.stdout: { 2024-03-25T23:02:38.591 INFO:tasks.workunit.client.0.smithi129.stdout: "id": 3, 2024-03-25T23:02:38.591 INFO:tasks.workunit.client.0.smithi129.stdout: "snapshots": [ 2024-03-25T23:02:38.591 INFO:tasks.workunit.client.0.smithi129.stdout: { 2024-03-25T23:02:38.591 INFO:tasks.workunit.client.0.smithi129.stdout: "id": 3, 2024-03-25T23:02:38.591 INFO:tasks.workunit.client.0.smithi129.stdout: "name": "snap13" 2024-03-25T23:02:38.592 INFO:tasks.workunit.client.0.smithi129.stdout: } 2024-03-25T23:02:38.592 INFO:tasks.workunit.client.0.smithi129.stdout: ], 2024-03-25T23:02:38.592 INFO:tasks.workunit.client.0.smithi129.stdout: "size": 4, 2024-03-25T23:02:38.592 INFO:tasks.workunit.client.0.smithi129.stdout: "overlaps": [] 2024-03-25T23:02:38.592 INFO:tasks.workunit.client.0.smithi129.stdout: }, 2024-03-25T23:02:38.592 INFO:tasks.workunit.client.0.smithi129.stdout: { 2024-03-25T23:02:38.592 INFO:tasks.workunit.client.0.smithi129.stdout: "id": 5, 2024-03-25T23:02:38.592 INFO:tasks.workunit.client.0.smithi129.stdout: "snapshots": [ 2024-03-25T23:02:38.592 INFO:tasks.workunit.client.0.smithi129.stdout: { 2024-03-25T23:02:38.592 INFO:tasks.workunit.client.0.smithi129.stdout: "id": 4, 2024-03-25T23:02:38.592 INFO:tasks.workunit.client.0.smithi129.stdout: "name": "snap24" 2024-03-25T23:02:38.592 INFO:tasks.workunit.client.0.smithi129.stdout: }, 2024-03-25T23:02:38.592 INFO:tasks.workunit.client.0.smithi129.stdout: { 2024-03-25T23:02:38.592 INFO:tasks.workunit.client.0.smithi129.stdout: "id": 5, 2024-03-25T23:02:38.592 INFO:tasks.workunit.client.0.smithi129.stdout: "name": "snap25" 2024-03-25T23:02:38.593 INFO:tasks.workunit.client.0.smithi129.stdout: } 2024-03-25T23:02:38.593 INFO:tasks.workunit.client.0.smithi129.stdout: ], 2024-03-25T23:02:38.593 INFO:tasks.workunit.client.0.smithi129.stdout: "size": 6, 2024-03-25T23:02:38.593 INFO:tasks.workunit.client.0.smithi129.stdout: "overlaps": [] 2024-03-25T23:02:38.593 INFO:tasks.workunit.client.0.smithi129.stdout: }, 2024-03-25T23:02:38.593 INFO:tasks.workunit.client.0.smithi129.stdout: { 2024-03-25T23:02:38.593 INFO:tasks.workunit.client.0.smithi129.stdout: "id": "head", 2024-03-25T23:02:38.593 INFO:tasks.workunit.client.0.smithi129.stdout: "snapshots": [], 2024-03-25T23:02:38.593 INFO:tasks.workunit.client.0.smithi129.stdout: "size": 6 2024-03-25T23:02:38.593 INFO:tasks.workunit.client.0.smithi129.stdout: } 2024-03-25T23:02:38.593 INFO:tasks.workunit.client.0.smithi129.stdout: ] 2024-03-25T23:02:38.593 INFO:tasks.workunit.client.0.smithi129.stdout:} 2024-03-25T23:02:38.594 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:80: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-03-25T23:02:38.716 INFO:tasks.workunit.client.0.smithi129.stdout:{ 2024-03-25T23:02:38.716 INFO:tasks.workunit.client.0.smithi129.stdout: "deep": true, 2024-03-25T23:02:38.716 INFO:tasks.workunit.client.0.smithi129.stdout: "must": true, 2024-03-25T23:02:38.717 INFO:tasks.workunit.client.0.smithi129.stdout: "stamp": "0.000000" 2024-03-25T23:02:38.717 INFO:tasks.workunit.client.0.smithi129.stdout:} 2024-03-25T23:02:38.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:84: TEST_truncated_sna_record: sleep 3 2024-03-25T23:02:41.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:85: TEST_truncated_sna_record: ceph pg dump pgs 2024-03-25T23:02:42.028 INFO:tasks.workunit.client.0.smithi129.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-25T23:02:42.028 INFO:tasks.workunit.client.0.smithi129.stdout:1.3 4 0 0 0 0 22 0 0 7 0 7 active+clean 2024-03-25T23:02:29.488050+0000 28'7 28:40 [1,2,0] 1 [1,2,0] 1 0'0 2024-03-25T23:02:28.045695+0000 0'0 2024-03-25T23:02:28.045695+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:02:28.045695+0000 0 0 2024-03-25T23:02:42.028 INFO:tasks.workunit.client.0.smithi129.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T23:02:29.760847+0000 0'0 28:32 [0,1,2] 0 [0,1,2] 0 0'0 2024-03-25T23:02:28.045695+0000 0'0 2024-03-25T23:02:28.045695+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:02:28.045695+0000 0 0 2024-03-25T23:02:42.028 INFO:tasks.workunit.client.0.smithi129.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T23:02:29.253998+0000 0'0 28:33 [2,0,1] 2 [2,0,1] 2 0'0 2024-03-25T23:02:28.045695+0000 0'0 2024-03-25T23:02:28.045695+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:02:28.045695+0000 0 0 2024-03-25T23:02:42.028 INFO:tasks.workunit.client.0.smithi129.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T23:02:29.489244+0000 0'0 28:35 [1,0,2] 1 [1,0,2] 1 0'0 2024-03-25T23:02:28.045695+0000 0'0 2024-03-25T23:02:28.045695+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:02:28.045695+0000 0 0 2024-03-25T23:02:42.028 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:02:42.028 INFO:tasks.workunit.client.0.smithi129.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-25T23:02:42.028 INFO:tasks.workunit.client.0.smithi129.stderr:dumped pgs 2024-03-25T23:02:42.044 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:86: TEST_truncated_sna_record: grep -a -q -- 'event: --^^^^---- ScrubFinished' td/osd-mapper/osd.1.log 2024-03-25T23:02:42.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:90: TEST_truncated_sna_record: ceph pg dump pgs 2024-03-25T23:02:42.335 INFO:tasks.workunit.client.0.smithi129.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-25T23:02:42.335 INFO:tasks.workunit.client.0.smithi129.stdout:1.3 4 0 0 0 0 22 0 0 7 0 7 active+clean 2024-03-25T23:02:29.488050+0000 28'7 28:40 [1,2,0] 1 [1,2,0] 1 0'0 2024-03-25T23:02:28.045695+0000 0'0 2024-03-25T23:02:28.045695+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:02:28.045695+0000 0 0 2024-03-25T23:02:42.335 INFO:tasks.workunit.client.0.smithi129.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T23:02:29.760847+0000 0'0 28:32 [0,1,2] 0 [0,1,2] 0 0'0 2024-03-25T23:02:28.045695+0000 0'0 2024-03-25T23:02:28.045695+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:02:28.045695+0000 0 0 2024-03-25T23:02:42.335 INFO:tasks.workunit.client.0.smithi129.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T23:02:29.253998+0000 0'0 28:33 [2,0,1] 2 [2,0,1] 2 0'0 2024-03-25T23:02:28.045695+0000 0'0 2024-03-25T23:02:28.045695+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:02:28.045695+0000 0 0 2024-03-25T23:02:42.335 INFO:tasks.workunit.client.0.smithi129.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T23:02:29.489244+0000 0'0 28:35 [1,0,2] 1 [1,0,2] 1 0'0 2024-03-25T23:02:28.045695+0000 0'0 2024-03-25T23:02:28.045695+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:02:28.045695+0000 0 0 2024-03-25T23:02:42.335 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:02:42.335 INFO:tasks.workunit.client.0.smithi129.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-25T23:02:42.336 INFO:tasks.workunit.client.0.smithi129.stderr:dumped pgs 2024-03-25T23:02:42.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:91: TEST_truncated_sna_record: ceph osd set noscrub 2024-03-25T23:02:42.750 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-25T23:02:42.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:92: TEST_truncated_sna_record: ceph osd set nodeep-scrub 2024-03-25T23:02:43.253 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-25T23:02:43.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:93: TEST_truncated_sna_record: sleep 5 2024-03-25T23:02:48.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:94: TEST_truncated_sna_record: grep -a -q -v ERR td/osd-mapper/osd.1.log 2024-03-25T23:02:48.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:97: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2024-03-25T23:02:48.281 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:02:48.281 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:02:48.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:02:48.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:02:48.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:02:48.593 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:02:48.594 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-25T23:02:48.594 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-03-25T23:02:48.622 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.621+0000 7fca0f561ac0 1 bdev(0x56242f75a000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-03-25T23:02:48.623 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.621+0000 7fca0f561ac0 1 bdev(0x56242f75a000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-25T23:02:48.623 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.621+0000 7fca0f561ac0 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-03-25T23:02:48.623 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.621+0000 7fca0f561ac0 1 bdev(0x56242f75a700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-03-25T23:02:48.623 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.621+0000 7fca0f561ac0 1 bdev(0x56242f75a700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-25T23:02:48.623 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.621+0000 7fca0f561ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-03-25T23:02:48.623 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.621+0000 7fca0f561ac0 1 bluefs mount 2024-03-25T23:02:48.635 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.633+0000 7fca0f561ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-25T23:02:48.655 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.653+0000 7fca0f561ac0 1 bluefs mount shared_bdev_used = 0 2024-03-25T23:02:48.655 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.653+0000 7fca0f561ac0 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-25T23:02:48.658 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.657+0000 7fca0f561ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-25T23:02:48.658 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.657+0000 7fca0f561ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-25T23:02:48.658 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.657+0000 7fca0f561ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-25T23:02:48.658 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.657+0000 7fca0f561ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-25T23:02:48.659 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.657+0000 7fca0f561ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-25T23:02:48.659 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.657+0000 7fca0f561ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-25T23:02:48.659 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.657+0000 7fca0f561ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-25T23:02:48.659 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.657+0000 7fca0f561ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-25T23:02:48.659 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.657+0000 7fca0f561ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-25T23:02:48.659 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.657+0000 7fca0f561ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-25T23:02:48.659 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.657+0000 7fca0f561ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-25T23:02:48.659 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.657+0000 7fca0f561ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-25T23:02:48.706 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.705+0000 7fca0f561ac0 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-25T23:02:48.707 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.705+0000 7fca0f561ac0 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2051 2024-03-25T23:02:48.707 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.705+0000 7fca0f561ac0 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2024-03-25T23:02:48.707 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.705+0000 7fca0f561ac0 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-03-25T23:02:48.707 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.705+0000 7fca0f561ac0 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2024-03-25T23:02:48.707 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.705+0000 7fca0f561ac0 1 freelist init 2024-03-25T23:02:48.707 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.705+0000 7fca0f561ac0 1 freelist _read_cfg 2024-03-25T23:02:48.707 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.705+0000 7fca0f561ac0 1 bluestore(td/osd-mapper/0) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-03-25T23:02:48.708 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.705+0000 7fca0f561ac0 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 25 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbc000, fragmentation 9.1553e-07 2024-03-25T23:02:48.708 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.705+0000 7fca0f561ac0 1 bluefs umount 2024-03-25T23:02:48.708 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.705+0000 7fca0f561ac0 1 bdev(0x56242f75a700 td/osd-mapper/0/block) close 2024-03-25T23:02:48.921 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.921+0000 7fca0f561ac0 1 bdev(0x56242f75a700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-03-25T23:02:48.922 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.921+0000 7fca0f561ac0 1 bdev(0x56242f75a700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-25T23:02:48.922 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.921+0000 7fca0f561ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-03-25T23:02:48.922 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.921+0000 7fca0f561ac0 1 bluefs mount 2024-03-25T23:02:48.922 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.921+0000 7fca0f561ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-25T23:02:48.928 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.925+0000 7fca0f561ac0 1 bluefs mount shared_bdev_used = 27459584 2024-03-25T23:02:48.928 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.925+0000 7fca0f561ac0 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-25T23:02:48.931 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.929+0000 7fca0f561ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-25T23:02:48.931 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.929+0000 7fca0f561ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-25T23:02:48.931 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.929+0000 7fca0f561ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-25T23:02:48.931 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.929+0000 7fca0f561ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-25T23:02:48.931 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.929+0000 7fca0f561ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-25T23:02:48.931 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.929+0000 7fca0f561ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-25T23:02:48.931 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.929+0000 7fca0f561ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-25T23:02:48.931 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.929+0000 7fca0f561ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-25T23:02:48.931 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.929+0000 7fca0f561ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-25T23:02:48.931 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.929+0000 7fca0f561ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-25T23:02:48.931 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.929+0000 7fca0f561ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-25T23:02:48.931 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.929+0000 7fca0f561ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-25T23:02:48.972 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.969+0000 7fca0f561ac0 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-25T23:02:48.973 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-03-25T23:02:48.973 INFO:tasks.workunit.client.0.smithi129.stdout:00000000 0b 05 58 02 00 00 99 b4 bc 82 c3 4e 48 95 95 ea |..X........NH...| 2024-03-25T23:02:48.974 INFO:tasks.workunit.client.0.smithi129.stdout:00000010 27 17 aa 85 e9 28 00 00 00 00 1e 00 00 00 00 00 |'....(..........| 2024-03-25T23:02:48.974 INFO:tasks.workunit.client.0.smithi129.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-25T23:02:48.974 INFO:tasks.workunit.client.0.smithi129.stdout:* 2024-03-25T23:02:48.974 INFO:tasks.workunit.client.0.smithi129.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-03-25T23:02:48.974 INFO:tasks.workunit.client.0.smithi129.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-03-25T23:02:48.974 INFO:tasks.workunit.client.0.smithi129.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-03-25T23:02:48.974 INFO:tasks.workunit.client.0.smithi129.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-03-25T23:02:48.974 INFO:tasks.workunit.client.0.smithi129.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-03-25T23:02:48.974 INFO:tasks.workunit.client.0.smithi129.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-03-25T23:02:48.974 INFO:tasks.workunit.client.0.smithi129.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-03-25T23:02:48.974 INFO:tasks.workunit.client.0.smithi129.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-03-25T23:02:48.974 INFO:tasks.workunit.client.0.smithi129.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-03-25T23:02:48.975 INFO:tasks.workunit.client.0.smithi129.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-03-25T23:02:48.975 INFO:tasks.workunit.client.0.smithi129.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-03-25T23:02:48.975 INFO:tasks.workunit.client.0.smithi129.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-03-25T23:02:48.975 INFO:tasks.workunit.client.0.smithi129.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-03-25T23:02:48.975 INFO:tasks.workunit.client.0.smithi129.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-03-25T23:02:48.975 INFO:tasks.workunit.client.0.smithi129.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-03-25T23:02:48.975 INFO:tasks.workunit.client.0.smithi129.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-03-25T23:02:48.975 INFO:tasks.workunit.client.0.smithi129.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-03-25T23:02:48.975 INFO:tasks.workunit.client.0.smithi129.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-03-25T23:02:48.975 INFO:tasks.workunit.client.0.smithi129.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-03-25T23:02:48.975 INFO:tasks.workunit.client.0.smithi129.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-03-25T23:02:48.975 INFO:tasks.workunit.client.0.smithi129.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-03-25T23:02:48.975 INFO:tasks.workunit.client.0.smithi129.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-03-25T23:02:48.975 INFO:tasks.workunit.client.0.smithi129.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-03-25T23:02:48.976 INFO:tasks.workunit.client.0.smithi129.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-03-25T23:02:48.976 INFO:tasks.workunit.client.0.smithi129.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-03-25T23:02:48.976 INFO:tasks.workunit.client.0.smithi129.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-03-25T23:02:48.976 INFO:tasks.workunit.client.0.smithi129.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-03-25T23:02:48.976 INFO:tasks.workunit.client.0.smithi129.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-03-25T23:02:48.976 INFO:tasks.workunit.client.0.smithi129.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-03-25T23:02:48.976 INFO:tasks.workunit.client.0.smithi129.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-03-25T23:02:48.976 INFO:tasks.workunit.client.0.smithi129.stdout:00000220 72 65 1e 00 00 00 05 00 00 00 d0 f8 b0 f8 17 5f |re............._| 2024-03-25T23:02:48.976 INFO:tasks.workunit.client.0.smithi129.stdout:00000230 43 ff 92 0c 01 1f 8b c0 c7 12 00 00 00 00 00 00 |C...............| 2024-03-25T23:02:48.976 INFO:tasks.workunit.client.0.smithi129.stdout:00000240 00 00 1e 00 00 00 38 02 02 66 53 de 40 17 01 00 |......8..fS.@...| 2024-03-25T23:02:48.976 INFO:tasks.workunit.client.0.smithi129.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1e 00 00 00 |..............| 2024-03-25T23:02:48.976 INFO:tasks.workunit.client.0.smithi129.stdout:0000025e 2024-03-25T23:02:48.976 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:02:48.977 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-03-25T23:02:48.977 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:02:48.977 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-03-25T23:02:48.977 INFO:tasks.workunit.client.0.smithi129.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-25T23:02:48.977 INFO:tasks.workunit.client.0.smithi129.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-03-25T23:02:48.977 INFO:tasks.workunit.client.0.smithi129.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-25T23:02:48.977 INFO:tasks.workunit.client.0.smithi129.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-03-25T23:02:48.977 INFO:tasks.workunit.client.0.smithi129.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-25T23:02:48.977 INFO:tasks.workunit.client.0.smithi129.stdout:00000047 2024-03-25T23:02:48.977 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:02:48.977 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-03-25T23:02:48.977 INFO:tasks.workunit.client.0.smithi129.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-03-25T23:02:48.977 INFO:tasks.workunit.client.0.smithi129.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-03-25T23:02:48.978 INFO:tasks.workunit.client.0.smithi129.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-25T23:02:48.978 INFO:tasks.workunit.client.0.smithi129.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-03-25T23:02:48.978 INFO:tasks.workunit.client.0.smithi129.stdout:0000003f 2024-03-25T23:02:48.978 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:02:48.978 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-03-25T23:02:48.978 INFO:tasks.workunit.client.0.smithi129.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-25T23:02:48.978 INFO:tasks.workunit.client.0.smithi129.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-03-25T23:02:48.978 INFO:tasks.workunit.client.0.smithi129.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-25T23:02:48.978 INFO:tasks.workunit.client.0.smithi129.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-03-25T23:02:48.978 INFO:tasks.workunit.client.0.smithi129.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-25T23:02:48.978 INFO:tasks.workunit.client.0.smithi129.stdout:00000047 2024-03-25T23:02:48.978 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:02:48.978 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-03-25T23:02:48.978 INFO:tasks.workunit.client.0.smithi129.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-25T23:02:48.979 INFO:tasks.workunit.client.0.smithi129.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-25T23:02:48.979 INFO:tasks.workunit.client.0.smithi129.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-25T23:02:48.979 INFO:tasks.workunit.client.0.smithi129.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-25T23:02:48.979 INFO:tasks.workunit.client.0.smithi129.stdout:0000003b 2024-03-25T23:02:48.979 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:02:48.979 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-03-25T23:02:48.979 INFO:tasks.workunit.client.0.smithi129.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-25T23:02:48.979 INFO:tasks.workunit.client.0.smithi129.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-25T23:02:48.979 INFO:tasks.workunit.client.0.smithi129.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-25T23:02:48.979 INFO:tasks.workunit.client.0.smithi129.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-25T23:02:48.979 INFO:tasks.workunit.client.0.smithi129.stdout:0000003b 2024-03-25T23:02:48.979 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:02:48.979 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-25T23:02:48.979 INFO:tasks.workunit.client.0.smithi129.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-25T23:02:48.980 INFO:tasks.workunit.client.0.smithi129.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-25T23:02:48.980 INFO:tasks.workunit.client.0.smithi129.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-25T23:02:48.980 INFO:tasks.workunit.client.0.smithi129.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-25T23:02:48.980 INFO:tasks.workunit.client.0.smithi129.stdout:0000003b 2024-03-25T23:02:48.980 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:02:48.980 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-03-25T23:02:48.980 INFO:tasks.workunit.client.0.smithi129.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-25T23:02:48.980 INFO:tasks.workunit.client.0.smithi129.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-25T23:02:48.980 INFO:tasks.workunit.client.0.smithi129.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-25T23:02:48.980 INFO:tasks.workunit.client.0.smithi129.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-25T23:02:48.980 INFO:tasks.workunit.client.0.smithi129.stdout:0000003b 2024-03-25T23:02:48.980 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:02:48.980 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-03-25T23:02:48.980 INFO:tasks.workunit.client.0.smithi129.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-25T23:02:48.981 INFO:tasks.workunit.client.0.smithi129.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-25T23:02:48.981 INFO:tasks.workunit.client.0.smithi129.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-25T23:02:48.981 INFO:tasks.workunit.client.0.smithi129.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-25T23:02:48.981 INFO:tasks.workunit.client.0.smithi129.stdout:0000003b 2024-03-25T23:02:48.981 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:02:48.981 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-03-25T23:02:48.981 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:02:48.981 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-03-25T23:02:48.981 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:02:48.981 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.969+0000 7fca0f561ac0 1 bluefs umount 2024-03-25T23:02:48.982 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:48.969+0000 7fca0f561ac0 1 bdev(0x56242f75a700 td/osd-mapper/0/block) close 2024-03-25T23:02:49.201 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.201+0000 7fca0f561ac0 1 freelist shutdown 2024-03-25T23:02:49.201 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.201+0000 7fca0f561ac0 1 bdev(0x56242f75a000 td/osd-mapper/0/block) close 2024-03-25T23:02:49.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-25T23:02:49.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-03-25T23:02:49.416 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: grep -a SNA_ 2024-03-25T23:02:49.444 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.441+0000 7fc63f8b7ac0 1 bdev(0x559906574000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-03-25T23:02:49.444 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.441+0000 7fc63f8b7ac0 1 bdev(0x559906574000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-25T23:02:49.444 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.441+0000 7fc63f8b7ac0 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-03-25T23:02:49.444 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.441+0000 7fc63f8b7ac0 1 bdev(0x559906574700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-03-25T23:02:49.444 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.441+0000 7fc63f8b7ac0 1 bdev(0x559906574700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-25T23:02:49.444 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.441+0000 7fc63f8b7ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-03-25T23:02:49.444 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.441+0000 7fc63f8b7ac0 1 bluefs mount 2024-03-25T23:02:49.458 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.457+0000 7fc63f8b7ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-25T23:02:49.466 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.465+0000 7fc63f8b7ac0 1 bluefs mount shared_bdev_used = 0 2024-03-25T23:02:49.467 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.465+0000 7fc63f8b7ac0 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-25T23:02:49.470 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.469+0000 7fc63f8b7ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-25T23:02:49.470 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.469+0000 7fc63f8b7ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-25T23:02:49.470 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.469+0000 7fc63f8b7ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-25T23:02:49.470 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.469+0000 7fc63f8b7ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-25T23:02:49.470 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.469+0000 7fc63f8b7ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-25T23:02:49.470 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.469+0000 7fc63f8b7ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-25T23:02:49.470 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.469+0000 7fc63f8b7ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-25T23:02:49.471 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.469+0000 7fc63f8b7ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-25T23:02:49.471 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.469+0000 7fc63f8b7ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-25T23:02:49.471 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.469+0000 7fc63f8b7ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-25T23:02:49.471 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.469+0000 7fc63f8b7ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-25T23:02:49.471 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.469+0000 7fc63f8b7ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-25T23:02:49.518 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.513+0000 7fc63f8b7ac0 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-25T23:02:49.518 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.513+0000 7fc63f8b7ac0 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2050 2024-03-25T23:02:49.518 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.513+0000 7fc63f8b7ac0 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2024-03-25T23:02:49.518 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.513+0000 7fc63f8b7ac0 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-03-25T23:02:49.518 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.513+0000 7fc63f8b7ac0 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2024-03-25T23:02:49.518 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.517+0000 7fc63f8b7ac0 1 freelist init 2024-03-25T23:02:49.518 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.517+0000 7fc63f8b7ac0 1 freelist _read_cfg 2024-03-25T23:02:49.518 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.517+0000 7fc63f8b7ac0 1 bluestore(td/osd-mapper/2) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-03-25T23:02:49.518 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.517+0000 7fc63f8b7ac0 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 15 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbc000, fragmentation 5.34059e-07 2024-03-25T23:02:49.519 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.517+0000 7fc63f8b7ac0 1 bluefs umount 2024-03-25T23:02:49.519 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.517+0000 7fc63f8b7ac0 1 bdev(0x559906574700 td/osd-mapper/2/block) close 2024-03-25T23:02:49.725 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.725+0000 7fc63f8b7ac0 1 bdev(0x559906574700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-03-25T23:02:49.725 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.725+0000 7fc63f8b7ac0 1 bdev(0x559906574700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-25T23:02:49.725 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.725+0000 7fc63f8b7ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-03-25T23:02:49.726 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.725+0000 7fc63f8b7ac0 1 bluefs mount 2024-03-25T23:02:49.733 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.733+0000 7fc63f8b7ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-25T23:02:49.739 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.737+0000 7fc63f8b7ac0 1 bluefs mount shared_bdev_used = 27459584 2024-03-25T23:02:49.739 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.737+0000 7fc63f8b7ac0 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-25T23:02:49.742 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.741+0000 7fc63f8b7ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-25T23:02:49.742 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.741+0000 7fc63f8b7ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-25T23:02:49.742 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.741+0000 7fc63f8b7ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-25T23:02:49.742 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.741+0000 7fc63f8b7ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-25T23:02:49.743 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.741+0000 7fc63f8b7ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-25T23:02:49.743 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.741+0000 7fc63f8b7ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-25T23:02:49.743 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.741+0000 7fc63f8b7ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-25T23:02:49.743 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.741+0000 7fc63f8b7ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-25T23:02:49.743 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.741+0000 7fc63f8b7ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-25T23:02:49.743 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.741+0000 7fc63f8b7ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-25T23:02:49.743 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.741+0000 7fc63f8b7ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-25T23:02:49.743 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.741+0000 7fc63f8b7ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-25T23:02:49.783 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.781+0000 7fc63f8b7ac0 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-25T23:02:49.783 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.781+0000 7fc63f8b7ac0 1 bluefs umount 2024-03-25T23:02:49.783 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:49.781+0000 7fc63f8b7ac0 1 bdev(0x559906574700 td/osd-mapper/2/block) close 2024-03-25T23:02:50.037 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:50.037+0000 7fc63f8b7ac0 1 freelist shutdown 2024-03-25T23:02:50.037 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:50.037+0000 7fc63f8b7ac0 1 bdev(0x559906574000 td/osd-mapper/2/block) close 2024-03-25T23:02:50.234 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-03-25T23:02:50.234 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-03-25T23:02:50.234 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-25T23:02:50.234 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-03-25T23:02:50.234 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-03-25T23:02:50.235 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-25T23:02:50.235 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: grep -a SNA_ /tmp/oo2.dump 2024-03-25T23:02:50.235 INFO:tasks.workunit.client.0.smithi129.stderr:grep: /tmp/oo2.dump: No such file or directory 2024-03-25T23:02:50.235 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-25T23:02:50.235 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-03-25T23:02:50.596 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-03-25T23:02:50.596 INFO:tasks.workunit.client.0.smithi129.stdout:00000000 0b 05 58 02 00 00 99 b4 bc 82 c3 4e 48 95 95 ea |..X........NH...| 2024-03-25T23:02:50.596 INFO:tasks.workunit.client.0.smithi129.stdout:00000010 27 17 aa 85 e9 28 02 00 00 00 1e 00 00 00 00 00 |'....(..........| 2024-03-25T23:02:50.596 INFO:tasks.workunit.client.0.smithi129.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-25T23:02:50.597 INFO:tasks.workunit.client.0.smithi129.stdout:* 2024-03-25T23:02:50.597 INFO:tasks.workunit.client.0.smithi129.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-03-25T23:02:50.597 INFO:tasks.workunit.client.0.smithi129.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-03-25T23:02:50.597 INFO:tasks.workunit.client.0.smithi129.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-03-25T23:02:50.597 INFO:tasks.workunit.client.0.smithi129.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-03-25T23:02:50.597 INFO:tasks.workunit.client.0.smithi129.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-03-25T23:02:50.597 INFO:tasks.workunit.client.0.smithi129.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-03-25T23:02:50.597 INFO:tasks.workunit.client.0.smithi129.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-03-25T23:02:50.597 INFO:tasks.workunit.client.0.smithi129.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-03-25T23:02:50.597 INFO:tasks.workunit.client.0.smithi129.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-03-25T23:02:50.597 INFO:tasks.workunit.client.0.smithi129.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-03-25T23:02:50.597 INFO:tasks.workunit.client.0.smithi129.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-03-25T23:02:50.597 INFO:tasks.workunit.client.0.smithi129.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-03-25T23:02:50.597 INFO:tasks.workunit.client.0.smithi129.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-03-25T23:02:50.598 INFO:tasks.workunit.client.0.smithi129.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-03-25T23:02:50.598 INFO:tasks.workunit.client.0.smithi129.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-03-25T23:02:50.598 INFO:tasks.workunit.client.0.smithi129.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-03-25T23:02:50.598 INFO:tasks.workunit.client.0.smithi129.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-03-25T23:02:50.598 INFO:tasks.workunit.client.0.smithi129.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-03-25T23:02:50.598 INFO:tasks.workunit.client.0.smithi129.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-03-25T23:02:50.598 INFO:tasks.workunit.client.0.smithi129.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-03-25T23:02:50.598 INFO:tasks.workunit.client.0.smithi129.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-03-25T23:02:50.598 INFO:tasks.workunit.client.0.smithi129.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-03-25T23:02:50.598 INFO:tasks.workunit.client.0.smithi129.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-03-25T23:02:50.598 INFO:tasks.workunit.client.0.smithi129.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-03-25T23:02:50.598 INFO:tasks.workunit.client.0.smithi129.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-03-25T23:02:50.598 INFO:tasks.workunit.client.0.smithi129.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-03-25T23:02:50.598 INFO:tasks.workunit.client.0.smithi129.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-03-25T23:02:50.598 INFO:tasks.workunit.client.0.smithi129.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-03-25T23:02:50.599 INFO:tasks.workunit.client.0.smithi129.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-03-25T23:02:50.599 INFO:tasks.workunit.client.0.smithi129.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-03-25T23:02:50.599 INFO:tasks.workunit.client.0.smithi129.stdout:00000220 72 65 1e 00 00 00 0f 00 00 00 6a 2a 0a 84 9b 88 |re........j*....| 2024-03-25T23:02:50.599 INFO:tasks.workunit.client.0.smithi129.stdout:00000230 4d bf ae 30 3c 6f 60 3f d6 76 00 00 00 00 00 00 |M..0= 3 )) 2024-03-25T23:02:51.881 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-03-25T23:02:51.909 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:51.905+0000 7f45b5f9fac0 1 bdev(0x555eeb9a6000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-03-25T23:02:51.909 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:51.905+0000 7f45b5f9fac0 1 bdev(0x555eeb9a6000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-25T23:02:51.909 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:51.905+0000 7f45b5f9fac0 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-03-25T23:02:51.909 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:51.905+0000 7f45b5f9fac0 1 bdev(0x555eeb9a6700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-03-25T23:02:51.909 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:51.905+0000 7f45b5f9fac0 1 bdev(0x555eeb9a6700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-25T23:02:51.909 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:51.905+0000 7f45b5f9fac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-03-25T23:02:51.909 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:51.905+0000 7f45b5f9fac0 1 bluefs mount 2024-03-25T23:02:51.909 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:51.909+0000 7f45b5f9fac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-25T23:02:51.916 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:51.913+0000 7f45b5f9fac0 1 bluefs mount shared_bdev_used = 0 2024-03-25T23:02:51.916 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:51.913+0000 7f45b5f9fac0 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-25T23:02:51.919 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:51.917+0000 7f45b5f9fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-25T23:02:51.920 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:51.917+0000 7f45b5f9fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-25T23:02:51.920 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:51.917+0000 7f45b5f9fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-25T23:02:51.920 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:51.917+0000 7f45b5f9fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-25T23:02:51.920 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:51.917+0000 7f45b5f9fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-25T23:02:51.920 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:51.917+0000 7f45b5f9fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-25T23:02:51.920 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:51.917+0000 7f45b5f9fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-25T23:02:51.920 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:51.917+0000 7f45b5f9fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-25T23:02:51.920 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:51.917+0000 7f45b5f9fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-25T23:02:51.920 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:51.917+0000 7f45b5f9fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-25T23:02:51.920 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:51.917+0000 7f45b5f9fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-25T23:02:51.920 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:51.917+0000 7f45b5f9fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-25T23:02:51.966 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:51.965+0000 7f45b5f9fac0 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-25T23:02:51.966 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:51.965+0000 7f45b5f9fac0 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2051 2024-03-25T23:02:51.966 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:51.965+0000 7f45b5f9fac0 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2024-03-25T23:02:51.967 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:51.965+0000 7f45b5f9fac0 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-03-25T23:02:51.967 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:51.965+0000 7f45b5f9fac0 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2024-03-25T23:02:51.967 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:51.965+0000 7f45b5f9fac0 1 freelist init 2024-03-25T23:02:51.967 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:51.965+0000 7f45b5f9fac0 1 freelist _read_cfg 2024-03-25T23:02:51.967 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:51.965+0000 7f45b5f9fac0 1 bluestore(td/osd-mapper/0) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-03-25T23:02:51.967 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:51.965+0000 7f45b5f9fac0 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 25 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbc000, fragmentation 9.1553e-07 2024-03-25T23:02:51.968 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:51.965+0000 7f45b5f9fac0 1 bluefs umount 2024-03-25T23:02:51.968 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:51.965+0000 7f45b5f9fac0 1 bdev(0x555eeb9a6700 td/osd-mapper/0/block) close 2024-03-25T23:02:52.189 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:52.189+0000 7f45b5f9fac0 1 bdev(0x555eeb9a6700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-03-25T23:02:52.189 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:52.189+0000 7f45b5f9fac0 1 bdev(0x555eeb9a6700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-25T23:02:52.189 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:52.189+0000 7f45b5f9fac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-03-25T23:02:52.190 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:52.189+0000 7f45b5f9fac0 1 bluefs mount 2024-03-25T23:02:52.190 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:52.189+0000 7f45b5f9fac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-25T23:02:52.196 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:52.193+0000 7f45b5f9fac0 1 bluefs mount shared_bdev_used = 27459584 2024-03-25T23:02:52.196 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:52.193+0000 7f45b5f9fac0 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-25T23:02:52.199 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:52.197+0000 7f45b5f9fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-25T23:02:52.199 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:52.197+0000 7f45b5f9fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-25T23:02:52.199 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:52.197+0000 7f45b5f9fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-25T23:02:52.199 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:52.197+0000 7f45b5f9fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-25T23:02:52.199 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:52.197+0000 7f45b5f9fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-25T23:02:52.199 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:52.197+0000 7f45b5f9fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-25T23:02:52.199 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:52.197+0000 7f45b5f9fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-25T23:02:52.199 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:52.197+0000 7f45b5f9fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-25T23:02:52.199 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:52.197+0000 7f45b5f9fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-25T23:02:52.199 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:52.197+0000 7f45b5f9fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-25T23:02:52.199 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:52.197+0000 7f45b5f9fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-25T23:02:52.200 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:52.197+0000 7f45b5f9fac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-25T23:02:52.239 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:52.237+0000 7f45b5f9fac0 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-25T23:02:52.241 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-03-25T23:02:52.241 INFO:tasks.workunit.client.0.smithi129.stdout:00000000 0b 05 58 02 00 00 99 b4 bc 82 c3 4e 48 95 95 ea |..X........NH...| 2024-03-25T23:02:52.241 INFO:tasks.workunit.client.0.smithi129.stdout:00000010 27 17 aa 85 e9 28 00 00 00 00 1e 00 00 00 00 00 |'....(..........| 2024-03-25T23:02:52.241 INFO:tasks.workunit.client.0.smithi129.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-25T23:02:52.241 INFO:tasks.workunit.client.0.smithi129.stdout:* 2024-03-25T23:02:52.241 INFO:tasks.workunit.client.0.smithi129.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-03-25T23:02:52.241 INFO:tasks.workunit.client.0.smithi129.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-03-25T23:02:52.241 INFO:tasks.workunit.client.0.smithi129.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-03-25T23:02:52.241 INFO:tasks.workunit.client.0.smithi129.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-03-25T23:02:52.241 INFO:tasks.workunit.client.0.smithi129.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-03-25T23:02:52.241 INFO:tasks.workunit.client.0.smithi129.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-03-25T23:02:52.241 INFO:tasks.workunit.client.0.smithi129.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-03-25T23:02:52.241 INFO:tasks.workunit.client.0.smithi129.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-03-25T23:02:52.242 INFO:tasks.workunit.client.0.smithi129.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-03-25T23:02:52.242 INFO:tasks.workunit.client.0.smithi129.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-03-25T23:02:52.242 INFO:tasks.workunit.client.0.smithi129.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-03-25T23:02:52.242 INFO:tasks.workunit.client.0.smithi129.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-03-25T23:02:52.242 INFO:tasks.workunit.client.0.smithi129.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-03-25T23:02:52.242 INFO:tasks.workunit.client.0.smithi129.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-03-25T23:02:52.242 INFO:tasks.workunit.client.0.smithi129.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-03-25T23:02:52.242 INFO:tasks.workunit.client.0.smithi129.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-03-25T23:02:52.242 INFO:tasks.workunit.client.0.smithi129.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-03-25T23:02:52.242 INFO:tasks.workunit.client.0.smithi129.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-03-25T23:02:52.242 INFO:tasks.workunit.client.0.smithi129.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-03-25T23:02:52.242 INFO:tasks.workunit.client.0.smithi129.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-03-25T23:02:52.242 INFO:tasks.workunit.client.0.smithi129.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-03-25T23:02:52.242 INFO:tasks.workunit.client.0.smithi129.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-03-25T23:02:52.243 INFO:tasks.workunit.client.0.smithi129.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-03-25T23:02:52.243 INFO:tasks.workunit.client.0.smithi129.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-03-25T23:02:52.243 INFO:tasks.workunit.client.0.smithi129.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-03-25T23:02:52.243 INFO:tasks.workunit.client.0.smithi129.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-03-25T23:02:52.243 INFO:tasks.workunit.client.0.smithi129.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-03-25T23:02:52.243 INFO:tasks.workunit.client.0.smithi129.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-03-25T23:02:52.243 INFO:tasks.workunit.client.0.smithi129.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-03-25T23:02:52.243 INFO:tasks.workunit.client.0.smithi129.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-03-25T23:02:52.243 INFO:tasks.workunit.client.0.smithi129.stdout:00000220 72 65 1e 00 00 00 05 00 00 00 d0 f8 b0 f8 17 5f |re............._| 2024-03-25T23:02:52.243 INFO:tasks.workunit.client.0.smithi129.stdout:00000230 43 ff 92 0c 01 1f 8b c0 c7 12 00 00 00 00 00 00 |C...............| 2024-03-25T23:02:52.243 INFO:tasks.workunit.client.0.smithi129.stdout:00000240 00 00 1e 00 00 00 38 02 02 66 53 de 40 17 01 00 |......8..fS.@...| 2024-03-25T23:02:52.243 INFO:tasks.workunit.client.0.smithi129.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1e 00 00 00 |..............| 2024-03-25T23:02:52.243 INFO:tasks.workunit.client.0.smithi129.stdout:0000025e 2024-03-25T23:02:52.244 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:02:52.244 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-03-25T23:02:52.244 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:02:52.244 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-03-25T23:02:52.244 INFO:tasks.workunit.client.0.smithi129.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-25T23:02:52.244 INFO:tasks.workunit.client.0.smithi129.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-03-25T23:02:52.244 INFO:tasks.workunit.client.0.smithi129.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-25T23:02:52.244 INFO:tasks.workunit.client.0.smithi129.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-03-25T23:02:52.244 INFO:tasks.workunit.client.0.smithi129.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-25T23:02:52.244 INFO:tasks.workunit.client.0.smithi129.stdout:00000047 2024-03-25T23:02:52.244 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:02:52.244 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-03-25T23:02:52.244 INFO:tasks.workunit.client.0.smithi129.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-03-25T23:02:52.245 INFO:tasks.workunit.client.0.smithi129.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-03-25T23:02:52.245 INFO:tasks.workunit.client.0.smithi129.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-25T23:02:52.245 INFO:tasks.workunit.client.0.smithi129.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-03-25T23:02:52.245 INFO:tasks.workunit.client.0.smithi129.stdout:0000003f 2024-03-25T23:02:52.245 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:02:52.245 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-03-25T23:02:52.245 INFO:tasks.workunit.client.0.smithi129.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-25T23:02:52.245 INFO:tasks.workunit.client.0.smithi129.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-03-25T23:02:52.245 INFO:tasks.workunit.client.0.smithi129.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-25T23:02:52.245 INFO:tasks.workunit.client.0.smithi129.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-03-25T23:02:52.245 INFO:tasks.workunit.client.0.smithi129.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-25T23:02:52.245 INFO:tasks.workunit.client.0.smithi129.stdout:00000047 2024-03-25T23:02:52.245 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:02:52.245 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-03-25T23:02:52.246 INFO:tasks.workunit.client.0.smithi129.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-25T23:02:52.246 INFO:tasks.workunit.client.0.smithi129.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-25T23:02:52.246 INFO:tasks.workunit.client.0.smithi129.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-25T23:02:52.246 INFO:tasks.workunit.client.0.smithi129.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-25T23:02:52.246 INFO:tasks.workunit.client.0.smithi129.stdout:0000003b 2024-03-25T23:02:52.246 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:02:52.246 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-03-25T23:02:52.246 INFO:tasks.workunit.client.0.smithi129.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-25T23:02:52.246 INFO:tasks.workunit.client.0.smithi129.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-25T23:02:52.246 INFO:tasks.workunit.client.0.smithi129.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-25T23:02:52.246 INFO:tasks.workunit.client.0.smithi129.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-25T23:02:52.246 INFO:tasks.workunit.client.0.smithi129.stdout:0000003b 2024-03-25T23:02:52.246 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:02:52.247 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-25T23:02:52.247 INFO:tasks.workunit.client.0.smithi129.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-25T23:02:52.247 INFO:tasks.workunit.client.0.smithi129.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-25T23:02:52.247 INFO:tasks.workunit.client.0.smithi129.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-25T23:02:52.247 INFO:tasks.workunit.client.0.smithi129.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-25T23:02:52.247 INFO:tasks.workunit.client.0.smithi129.stdout:0000003b 2024-03-25T23:02:52.247 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:02:52.247 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-03-25T23:02:52.247 INFO:tasks.workunit.client.0.smithi129.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-25T23:02:52.247 INFO:tasks.workunit.client.0.smithi129.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-25T23:02:52.247 INFO:tasks.workunit.client.0.smithi129.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-25T23:02:52.247 INFO:tasks.workunit.client.0.smithi129.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-25T23:02:52.248 INFO:tasks.workunit.client.0.smithi129.stdout:0000003b 2024-03-25T23:02:52.248 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:02:52.248 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-03-25T23:02:52.248 INFO:tasks.workunit.client.0.smithi129.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-25T23:02:52.248 INFO:tasks.workunit.client.0.smithi129.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-25T23:02:52.248 INFO:tasks.workunit.client.0.smithi129.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-25T23:02:52.248 INFO:tasks.workunit.client.0.smithi129.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-25T23:02:52.248 INFO:tasks.workunit.client.0.smithi129.stdout:0000003b 2024-03-25T23:02:52.248 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:02:52.248 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-03-25T23:02:52.248 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:02:52.248 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-03-25T23:02:52.248 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:02:52.249 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:52.237+0000 7f45b5f9fac0 1 bluefs umount 2024-03-25T23:02:52.249 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:52.237+0000 7f45b5f9fac0 1 bdev(0x555eeb9a6700 td/osd-mapper/0/block) close 2024-03-25T23:02:52.469 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:52.469+0000 7f45b5f9fac0 1 freelist shutdown 2024-03-25T23:02:52.469 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:52.469+0000 7f45b5f9fac0 1 bdev(0x555eeb9a6000 td/osd-mapper/0/block) close 2024-03-25T23:02:52.716 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-03-25T23:02:52.717 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-03-25T23:02:52.717 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-03-25T23:02:53.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-25T23:02:53.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-25T23:02:53.533 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-03-25T23:02:53.533 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-03-25T23:02:53.533 INFO:tasks.workunit.client.0.smithi129.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-25T23:02:53.534 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-03-25T23:02:53.535 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.eJLm70RKJb_the_val 2024-03-25T23:02:53.535 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-25T23:02:53.535 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.eJLm70RKJb_the_val' 2024-03-25T23:02:53.536 INFO:tasks.workunit.client.0.smithi129.stdout:Value dumped in: /tmp/tmp.eJLm70RKJb_the_val 2024-03-25T23:02:53.536 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.eJLm70RKJb_the_val 2024-03-25T23:02:53.898 INFO:tasks.workunit.client.0.smithi129.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-03-25T23:02:54.363 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-25T23:02:54.363 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.eJLm70RKJb_the_val 2024-03-25T23:02:54.370 INFO:tasks.workunit.client.0.smithi129.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-03-25T23:02:54.370 INFO:tasks.workunit.client.0.smithi129.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-03-25T23:02:54.370 INFO:tasks.workunit.client.0.smithi129.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-03-25T23:02:54.370 INFO:tasks.workunit.client.0.smithi129.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-03-25T23:02:54.370 INFO:tasks.workunit.client.0.smithi129.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-03-25T23:02:54.370 INFO:tasks.workunit.client.0.smithi129.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 � u � 205 \0 \0 2024-03-25T23:02:54.370 INFO:tasks.workunit.client.0.smithi129.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-03-25T23:02:54.370 INFO:tasks.workunit.client.0.smithi129.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-03-25T23:02:54.370 INFO:tasks.workunit.client.0.smithi129.stdout:0000073 2024-03-25T23:02:54.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-03-25T23:02:54.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-25T23:02:55.941 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.eJLm70RKJb_the_val 2024-03-25T23:02:57.009 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-25T23:02:57.009 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-25T23:02:57.009 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2024-03-25T23:02:57.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1)' 2024-03-25T23:02:57.010 INFO:tasks.workunit.client.0.smithi129.stdout:corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1) 2024-03-25T23:02:57.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-03-25T23:02:57.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-03-25T23:02:57.040 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.037+0000 7f4fb71bcac0 1 bdev(0x55fd67768000 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-03-25T23:02:57.040 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.037+0000 7f4fb71bcac0 1 bdev(0x55fd67768000 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-25T23:02:57.040 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.037+0000 7f4fb71bcac0 1 bluestore(td/osd-mapper/1) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-03-25T23:02:57.040 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.037+0000 7f4fb71bcac0 1 bdev(0x55fd67768700 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-03-25T23:02:57.040 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.037+0000 7f4fb71bcac0 1 bdev(0x55fd67768700 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-25T23:02:57.040 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.037+0000 7f4fb71bcac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2024-03-25T23:02:57.040 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.037+0000 7f4fb71bcac0 1 bluefs mount 2024-03-25T23:02:57.044 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.041+0000 7f4fb71bcac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-25T23:02:57.053 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.049+0000 7f4fb71bcac0 1 bluefs mount shared_bdev_used = 0 2024-03-25T23:02:57.053 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.049+0000 7f4fb71bcac0 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-25T23:02:57.056 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.053+0000 7f4fb71bcac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-25T23:02:57.056 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.053+0000 7f4fb71bcac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-25T23:02:57.056 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.053+0000 7f4fb71bcac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-25T23:02:57.056 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.053+0000 7f4fb71bcac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-25T23:02:57.056 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.053+0000 7f4fb71bcac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-25T23:02:57.056 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.053+0000 7f4fb71bcac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-25T23:02:57.056 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.053+0000 7f4fb71bcac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-25T23:02:57.057 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.053+0000 7f4fb71bcac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-25T23:02:57.057 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.053+0000 7f4fb71bcac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-25T23:02:57.057 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.053+0000 7f4fb71bcac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-25T23:02:57.057 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.053+0000 7f4fb71bcac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-25T23:02:57.057 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.053+0000 7f4fb71bcac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-25T23:02:57.102 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.101+0000 7f4fb71bcac0 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-25T23:02:57.103 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.101+0000 7f4fb71bcac0 1 bluestore(td/osd-mapper/1) _open_super_meta old nid_max 2050 2024-03-25T23:02:57.103 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.101+0000 7f4fb71bcac0 1 bluestore(td/osd-mapper/1) _open_super_meta old blobid_max 20480 2024-03-25T23:02:57.103 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.101+0000 7f4fb71bcac0 1 bluestore(td/osd-mapper/1) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-03-25T23:02:57.103 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.101+0000 7f4fb71bcac0 1 bluestore(td/osd-mapper/1) _open_super_meta min_alloc_size 0x1000 2024-03-25T23:02:57.103 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.101+0000 7f4fb71bcac0 1 freelist init 2024-03-25T23:02:57.103 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.101+0000 7f4fb71bcac0 1 freelist _read_cfg 2024-03-25T23:02:57.103 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.101+0000 7f4fb71bcac0 1 bluestore(td/osd-mapper/1) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-03-25T23:02:57.104 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.101+0000 7f4fb71bcac0 1 bluestore(td/osd-mapper/1) _init_alloc loaded 100 GiB in 20 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbc000, fragmentation 7.24794e-07 2024-03-25T23:02:57.104 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.101+0000 7f4fb71bcac0 1 bluefs umount 2024-03-25T23:02:57.104 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.101+0000 7f4fb71bcac0 1 bdev(0x55fd67768700 td/osd-mapper/1/block) close 2024-03-25T23:02:57.321 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.321+0000 7f4fb71bcac0 1 bdev(0x55fd67768700 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-03-25T23:02:57.321 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.321+0000 7f4fb71bcac0 1 bdev(0x55fd67768700 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-25T23:02:57.321 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.321+0000 7f4fb71bcac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2024-03-25T23:02:57.322 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.321+0000 7f4fb71bcac0 1 bluefs mount 2024-03-25T23:02:57.328 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.325+0000 7f4fb71bcac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-25T23:02:57.336 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.333+0000 7f4fb71bcac0 1 bluefs mount shared_bdev_used = 27459584 2024-03-25T23:02:57.336 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.333+0000 7f4fb71bcac0 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-25T23:02:57.339 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.337+0000 7f4fb71bcac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-25T23:02:57.339 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.337+0000 7f4fb71bcac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-25T23:02:57.339 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.337+0000 7f4fb71bcac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-25T23:02:57.339 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.337+0000 7f4fb71bcac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-25T23:02:57.339 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.337+0000 7f4fb71bcac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-25T23:02:57.339 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.337+0000 7f4fb71bcac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-25T23:02:57.339 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.337+0000 7f4fb71bcac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-25T23:02:57.339 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.337+0000 7f4fb71bcac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-25T23:02:57.339 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.337+0000 7f4fb71bcac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-25T23:02:57.339 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.337+0000 7f4fb71bcac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-25T23:02:57.340 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.337+0000 7f4fb71bcac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-25T23:02:57.340 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.337+0000 7f4fb71bcac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-25T23:02:57.378 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.377+0000 7f4fb71bcac0 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-25T23:02:57.379 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-03-25T23:02:57.379 INFO:tasks.workunit.client.0.smithi129.stdout:00000000 0b 05 58 02 00 00 99 b4 bc 82 c3 4e 48 95 95 ea |..X........NH...| 2024-03-25T23:02:57.379 INFO:tasks.workunit.client.0.smithi129.stdout:00000010 27 17 aa 85 e9 28 01 00 00 00 1e 00 00 00 00 00 |'....(..........| 2024-03-25T23:02:57.379 INFO:tasks.workunit.client.0.smithi129.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-25T23:02:57.380 INFO:tasks.workunit.client.0.smithi129.stdout:* 2024-03-25T23:02:57.380 INFO:tasks.workunit.client.0.smithi129.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-03-25T23:02:57.380 INFO:tasks.workunit.client.0.smithi129.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-03-25T23:02:57.380 INFO:tasks.workunit.client.0.smithi129.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-03-25T23:02:57.380 INFO:tasks.workunit.client.0.smithi129.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-03-25T23:02:57.380 INFO:tasks.workunit.client.0.smithi129.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-03-25T23:02:57.380 INFO:tasks.workunit.client.0.smithi129.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-03-25T23:02:57.380 INFO:tasks.workunit.client.0.smithi129.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-03-25T23:02:57.380 INFO:tasks.workunit.client.0.smithi129.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-03-25T23:02:57.380 INFO:tasks.workunit.client.0.smithi129.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-03-25T23:02:57.380 INFO:tasks.workunit.client.0.smithi129.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-03-25T23:02:57.380 INFO:tasks.workunit.client.0.smithi129.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-03-25T23:02:57.380 INFO:tasks.workunit.client.0.smithi129.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-03-25T23:02:57.380 INFO:tasks.workunit.client.0.smithi129.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-03-25T23:02:57.381 INFO:tasks.workunit.client.0.smithi129.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-03-25T23:02:57.381 INFO:tasks.workunit.client.0.smithi129.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-03-25T23:02:57.381 INFO:tasks.workunit.client.0.smithi129.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-03-25T23:02:57.381 INFO:tasks.workunit.client.0.smithi129.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-03-25T23:02:57.381 INFO:tasks.workunit.client.0.smithi129.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-03-25T23:02:57.381 INFO:tasks.workunit.client.0.smithi129.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-03-25T23:02:57.381 INFO:tasks.workunit.client.0.smithi129.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-03-25T23:02:57.381 INFO:tasks.workunit.client.0.smithi129.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-03-25T23:02:57.381 INFO:tasks.workunit.client.0.smithi129.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-03-25T23:02:57.381 INFO:tasks.workunit.client.0.smithi129.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-03-25T23:02:57.381 INFO:tasks.workunit.client.0.smithi129.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-03-25T23:02:57.381 INFO:tasks.workunit.client.0.smithi129.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-03-25T23:02:57.381 INFO:tasks.workunit.client.0.smithi129.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-03-25T23:02:57.381 INFO:tasks.workunit.client.0.smithi129.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-03-25T23:02:57.382 INFO:tasks.workunit.client.0.smithi129.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-03-25T23:02:57.382 INFO:tasks.workunit.client.0.smithi129.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-03-25T23:02:57.382 INFO:tasks.workunit.client.0.smithi129.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-03-25T23:02:57.382 INFO:tasks.workunit.client.0.smithi129.stdout:00000220 72 65 1e 00 00 00 0a 00 00 00 7e f9 1d 1c 2b f1 |re........~...+.| 2024-03-25T23:02:57.382 INFO:tasks.workunit.client.0.smithi129.stdout:00000230 4f 0c 86 db 44 d8 43 c5 75 5b 00 00 00 00 00 00 |O...D.C.u[......| 2024-03-25T23:02:57.382 INFO:tasks.workunit.client.0.smithi129.stdout:00000240 00 00 1e 00 00 00 5b 02 02 66 d8 5c f3 2f 01 00 |......[..f.\./..| 2024-03-25T23:02:57.382 INFO:tasks.workunit.client.0.smithi129.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1e 00 00 00 |..............| 2024-03-25T23:02:57.382 INFO:tasks.workunit.client.0.smithi129.stdout:0000025e 2024-03-25T23:02:57.382 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:02:57.382 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-03-25T23:02:57.382 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:02:57.382 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-03-25T23:02:57.382 INFO:tasks.workunit.client.0.smithi129.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-25T23:02:57.383 INFO:tasks.workunit.client.0.smithi129.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-03-25T23:02:57.383 INFO:tasks.workunit.client.0.smithi129.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-25T23:02:57.383 INFO:tasks.workunit.client.0.smithi129.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-03-25T23:02:57.383 INFO:tasks.workunit.client.0.smithi129.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-25T23:02:57.383 INFO:tasks.workunit.client.0.smithi129.stdout:00000047 2024-03-25T23:02:57.383 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:02:57.383 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-03-25T23:02:57.383 INFO:tasks.workunit.client.0.smithi129.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-03-25T23:02:57.383 INFO:tasks.workunit.client.0.smithi129.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-03-25T23:02:57.383 INFO:tasks.workunit.client.0.smithi129.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-25T23:02:57.383 INFO:tasks.workunit.client.0.smithi129.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-03-25T23:02:57.383 INFO:tasks.workunit.client.0.smithi129.stdout:0000003f 2024-03-25T23:02:57.383 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:02:57.383 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-03-25T23:02:57.383 INFO:tasks.workunit.client.0.smithi129.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-25T23:02:57.384 INFO:tasks.workunit.client.0.smithi129.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-03-25T23:02:57.384 INFO:tasks.workunit.client.0.smithi129.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-25T23:02:57.384 INFO:tasks.workunit.client.0.smithi129.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-03-25T23:02:57.384 INFO:tasks.workunit.client.0.smithi129.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-25T23:02:57.384 INFO:tasks.workunit.client.0.smithi129.stdout:00000047 2024-03-25T23:02:57.384 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:02:57.384 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-03-25T23:02:57.384 INFO:tasks.workunit.client.0.smithi129.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-25T23:02:57.384 INFO:tasks.workunit.client.0.smithi129.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-25T23:02:57.384 INFO:tasks.workunit.client.0.smithi129.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-25T23:02:57.384 INFO:tasks.workunit.client.0.smithi129.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-25T23:02:57.384 INFO:tasks.workunit.client.0.smithi129.stdout:0000003b 2024-03-25T23:02:57.384 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:02:57.384 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-03-25T23:02:57.385 INFO:tasks.workunit.client.0.smithi129.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-25T23:02:57.385 INFO:tasks.workunit.client.0.smithi129.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-25T23:02:57.385 INFO:tasks.workunit.client.0.smithi129.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-25T23:02:57.385 INFO:tasks.workunit.client.0.smithi129.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-25T23:02:57.385 INFO:tasks.workunit.client.0.smithi129.stdout:0000003b 2024-03-25T23:02:57.385 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:02:57.385 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-25T23:02:57.385 INFO:tasks.workunit.client.0.smithi129.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-25T23:02:57.385 INFO:tasks.workunit.client.0.smithi129.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-25T23:02:57.385 INFO:tasks.workunit.client.0.smithi129.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-25T23:02:57.385 INFO:tasks.workunit.client.0.smithi129.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-25T23:02:57.385 INFO:tasks.workunit.client.0.smithi129.stdout:0000003b 2024-03-25T23:02:57.385 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:02:57.385 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-03-25T23:02:57.385 INFO:tasks.workunit.client.0.smithi129.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-25T23:02:57.386 INFO:tasks.workunit.client.0.smithi129.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-25T23:02:57.386 INFO:tasks.workunit.client.0.smithi129.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-25T23:02:57.386 INFO:tasks.workunit.client.0.smithi129.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-25T23:02:57.386 INFO:tasks.workunit.client.0.smithi129.stdout:0000003b 2024-03-25T23:02:57.386 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:02:57.386 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-03-25T23:02:57.386 INFO:tasks.workunit.client.0.smithi129.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-25T23:02:57.386 INFO:tasks.workunit.client.0.smithi129.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-25T23:02:57.386 INFO:tasks.workunit.client.0.smithi129.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-25T23:02:57.386 INFO:tasks.workunit.client.0.smithi129.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-25T23:02:57.386 INFO:tasks.workunit.client.0.smithi129.stdout:0000003b 2024-03-25T23:02:57.386 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:02:57.386 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-03-25T23:02:57.386 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:02:57.386 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-03-25T23:02:57.387 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:02:57.387 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.377+0000 7f4fb71bcac0 1 bluefs umount 2024-03-25T23:02:57.387 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.377+0000 7f4fb71bcac0 1 bdev(0x55fd67768700 td/osd-mapper/1/block) close 2024-03-25T23:02:57.609 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.609+0000 7f4fb71bcac0 1 freelist shutdown 2024-03-25T23:02:57.609 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:02:57.609+0000 7f4fb71bcac0 1 bdev(0x55fd67768000 td/osd-mapper/1/block) close 2024-03-25T23:02:57.828 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-03-25T23:02:57.828 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-03-25T23:02:57.828 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-03-25T23:02:58.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-25T23:02:58.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-25T23:02:58.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-03-25T23:02:58.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-03-25T23:02:58.649 INFO:tasks.workunit.client.0.smithi129.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-25T23:02:58.650 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-03-25T23:02:58.652 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.lFqjeDG6PE_the_val 2024-03-25T23:02:58.652 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-25T23:02:58.652 INFO:tasks.workunit.client.0.smithi129.stdout:Value dumped in: /tmp/tmp.lFqjeDG6PE_the_val 2024-03-25T23:02:58.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.lFqjeDG6PE_the_val' 2024-03-25T23:02:58.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.lFqjeDG6PE_the_val 2024-03-25T23:02:59.030 INFO:tasks.workunit.client.0.smithi129.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-03-25T23:02:59.472 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-25T23:02:59.472 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.lFqjeDG6PE_the_val 2024-03-25T23:02:59.473 INFO:tasks.workunit.client.0.smithi129.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-03-25T23:02:59.473 INFO:tasks.workunit.client.0.smithi129.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-03-25T23:02:59.473 INFO:tasks.workunit.client.0.smithi129.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-03-25T23:02:59.473 INFO:tasks.workunit.client.0.smithi129.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-03-25T23:02:59.474 INFO:tasks.workunit.client.0.smithi129.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-03-25T23:02:59.474 INFO:tasks.workunit.client.0.smithi129.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 � u � 205 \0 \0 2024-03-25T23:02:59.474 INFO:tasks.workunit.client.0.smithi129.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-03-25T23:02:59.474 INFO:tasks.workunit.client.0.smithi129.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-03-25T23:02:59.474 INFO:tasks.workunit.client.0.smithi129.stdout:0000073 2024-03-25T23:02:59.475 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-03-25T23:02:59.475 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-25T23:03:01.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.lFqjeDG6PE_the_val 2024-03-25T23:03:02.118 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-25T23:03:02.118 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-25T23:03:02.118 INFO:tasks.workunit.client.0.smithi129.stdout:corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2) 2024-03-25T23:03:02.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/2 2024-03-25T23:03:02.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2)' 2024-03-25T23:03:02.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-03-25T23:03:02.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-03-25T23:03:02.145 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.141+0000 7f14f97b9ac0 1 bdev(0x556387e1a000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-03-25T23:03:02.145 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.141+0000 7f14f97b9ac0 1 bdev(0x556387e1a000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-25T23:03:02.145 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.141+0000 7f14f97b9ac0 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-03-25T23:03:02.145 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.141+0000 7f14f97b9ac0 1 bdev(0x556387e1a700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-03-25T23:03:02.145 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.141+0000 7f14f97b9ac0 1 bdev(0x556387e1a700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-25T23:03:02.145 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.141+0000 7f14f97b9ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-03-25T23:03:02.145 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.141+0000 7f14f97b9ac0 1 bluefs mount 2024-03-25T23:03:02.156 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.153+0000 7f14f97b9ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-25T23:03:02.165 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.161+0000 7f14f97b9ac0 1 bluefs mount shared_bdev_used = 0 2024-03-25T23:03:02.165 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.165+0000 7f14f97b9ac0 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-25T23:03:02.168 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.165+0000 7f14f97b9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-25T23:03:02.168 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.165+0000 7f14f97b9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-25T23:03:02.168 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.165+0000 7f14f97b9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-25T23:03:02.168 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.165+0000 7f14f97b9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-25T23:03:02.169 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.165+0000 7f14f97b9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-25T23:03:02.169 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.165+0000 7f14f97b9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-25T23:03:02.169 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.165+0000 7f14f97b9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-25T23:03:02.169 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.165+0000 7f14f97b9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-25T23:03:02.169 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.165+0000 7f14f97b9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-25T23:03:02.169 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.165+0000 7f14f97b9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-25T23:03:02.169 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.165+0000 7f14f97b9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-25T23:03:02.169 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.165+0000 7f14f97b9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-25T23:03:02.214 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.213+0000 7f14f97b9ac0 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-25T23:03:02.215 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.213+0000 7f14f97b9ac0 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2050 2024-03-25T23:03:02.215 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.213+0000 7f14f97b9ac0 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2024-03-25T23:03:02.215 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.213+0000 7f14f97b9ac0 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-03-25T23:03:02.215 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.213+0000 7f14f97b9ac0 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2024-03-25T23:03:02.215 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.213+0000 7f14f97b9ac0 1 freelist init 2024-03-25T23:03:02.215 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.213+0000 7f14f97b9ac0 1 freelist _read_cfg 2024-03-25T23:03:02.215 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.213+0000 7f14f97b9ac0 1 bluestore(td/osd-mapper/2) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-03-25T23:03:02.216 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.213+0000 7f14f97b9ac0 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 15 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbc000, fragmentation 5.34059e-07 2024-03-25T23:03:02.216 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.213+0000 7f14f97b9ac0 1 bluefs umount 2024-03-25T23:03:02.216 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.213+0000 7f14f97b9ac0 1 bdev(0x556387e1a700 td/osd-mapper/2/block) close 2024-03-25T23:03:02.425 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.425+0000 7f14f97b9ac0 1 bdev(0x556387e1a700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-03-25T23:03:02.426 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.425+0000 7f14f97b9ac0 1 bdev(0x556387e1a700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-25T23:03:02.426 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.425+0000 7f14f97b9ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-03-25T23:03:02.426 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.425+0000 7f14f97b9ac0 1 bluefs mount 2024-03-25T23:03:02.426 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.425+0000 7f14f97b9ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-25T23:03:02.432 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.429+0000 7f14f97b9ac0 1 bluefs mount shared_bdev_used = 27459584 2024-03-25T23:03:02.432 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.429+0000 7f14f97b9ac0 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-25T23:03:02.435 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.433+0000 7f14f97b9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-25T23:03:02.435 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.433+0000 7f14f97b9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-25T23:03:02.435 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.433+0000 7f14f97b9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-25T23:03:02.435 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.433+0000 7f14f97b9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-25T23:03:02.435 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.433+0000 7f14f97b9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-25T23:03:02.435 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.433+0000 7f14f97b9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-25T23:03:02.435 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.433+0000 7f14f97b9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-25T23:03:02.435 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.433+0000 7f14f97b9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-25T23:03:02.435 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.433+0000 7f14f97b9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-25T23:03:02.435 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.433+0000 7f14f97b9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-25T23:03:02.435 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.433+0000 7f14f97b9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-25T23:03:02.435 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.433+0000 7f14f97b9ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-25T23:03:02.474 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:02.473+0000 7f14f97b9ac0 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-25T23:03:02.475 INFO:tasks.workunit.client.0.smithi129.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-03-25T23:03:02.475 INFO:tasks.workunit.client.0.smithi129.stdout:00000000 0b 05 58 02 00 00 99 b4 bc 82 c3 4e 48 95 95 ea |..X........NH...| 2024-03-25T23:03:02.475 INFO:tasks.workunit.client.0.smithi129.stdout:00000010 27 17 aa 85 e9 28 02 00 00 00 1e 00 00 00 00 00 |'....(..........| 2024-03-25T23:03:02.475 INFO:tasks.workunit.client.0.smithi129.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-25T23:03:02.475 INFO:tasks.workunit.client.0.smithi129.stdout:* 2024-03-25T23:03:02.475 INFO:tasks.workunit.client.0.smithi129.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-03-25T23:03:02.475 INFO:tasks.workunit.client.0.smithi129.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-03-25T23:03:02.475 INFO:tasks.workunit.client.0.smithi129.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-03-25T23:03:02.475 INFO:tasks.workunit.client.0.smithi129.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-03-25T23:03:02.475 INFO:tasks.workunit.client.0.smithi129.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-03-25T23:03:02.476 INFO:tasks.workunit.client.0.smithi129.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-03-25T23:03:02.476 INFO:tasks.workunit.client.0.smithi129.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-03-25T23:03:02.476 INFO:tasks.workunit.client.0.smithi129.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-03-25T23:03:02.476 INFO:tasks.workunit.client.0.smithi129.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-03-25T23:03:02.476 INFO:tasks.workunit.client.0.smithi129.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-03-25T23:03:02.476 INFO:tasks.workunit.client.0.smithi129.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-03-25T23:03:02.476 INFO:tasks.workunit.client.0.smithi129.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-03-25T23:03:02.476 INFO:tasks.workunit.client.0.smithi129.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-03-25T23:03:02.476 INFO:tasks.workunit.client.0.smithi129.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-03-25T23:03:02.476 INFO:tasks.workunit.client.0.smithi129.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-03-25T23:03:02.476 INFO:tasks.workunit.client.0.smithi129.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-03-25T23:03:02.476 INFO:tasks.workunit.client.0.smithi129.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-03-25T23:03:02.476 INFO:tasks.workunit.client.0.smithi129.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-03-25T23:03:02.476 INFO:tasks.workunit.client.0.smithi129.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-03-25T23:03:02.476 INFO:tasks.workunit.client.0.smithi129.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-03-25T23:03:02.477 INFO:tasks.workunit.client.0.smithi129.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-03-25T23:03:02.477 INFO:tasks.workunit.client.0.smithi129.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-03-25T23:03:02.477 INFO:tasks.workunit.client.0.smithi129.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-03-25T23:03:02.477 INFO:tasks.workunit.client.0.smithi129.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-03-25T23:03:02.477 INFO:tasks.workunit.client.0.smithi129.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-03-25T23:03:02.477 INFO:tasks.workunit.client.0.smithi129.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-03-25T23:03:02.477 INFO:tasks.workunit.client.0.smithi129.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-03-25T23:03:02.477 INFO:tasks.workunit.client.0.smithi129.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-03-25T23:03:02.477 INFO:tasks.workunit.client.0.smithi129.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-03-25T23:03:02.477 INFO:tasks.workunit.client.0.smithi129.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-03-25T23:03:02.477 INFO:tasks.workunit.client.0.smithi129.stdout:00000220 72 65 1e 00 00 00 0f 00 00 00 6a 2a 0a 84 9b 88 |re........j*....| 2024-03-25T23:03:02.477 INFO:tasks.workunit.client.0.smithi129.stdout:00000230 4d bf ae 30 3c 6f 60 3f d6 76 00 00 00 00 00 00 |M..0= 1 )) 2024-03-25T23:03:03.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-03-25T23:03:03.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-03-25T23:03:03.758 INFO:tasks.workunit.client.0.smithi129.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-25T23:03:03.758 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-03-25T23:03:03.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.RNQnos9gJ5_the_val 2024-03-25T23:03:03.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-25T23:03:03.760 INFO:tasks.workunit.client.0.smithi129.stdout:Value dumped in: /tmp/tmp.RNQnos9gJ5_the_val 2024-03-25T23:03:03.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.RNQnos9gJ5_the_val' 2024-03-25T23:03:03.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.RNQnos9gJ5_the_val 2024-03-25T23:03:04.134 INFO:tasks.workunit.client.0.smithi129.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-03-25T23:03:04.583 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-25T23:03:04.583 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.RNQnos9gJ5_the_val 2024-03-25T23:03:04.585 INFO:tasks.workunit.client.0.smithi129.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-03-25T23:03:04.585 INFO:tasks.workunit.client.0.smithi129.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-03-25T23:03:04.585 INFO:tasks.workunit.client.0.smithi129.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-03-25T23:03:04.585 INFO:tasks.workunit.client.0.smithi129.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-03-25T23:03:04.585 INFO:tasks.workunit.client.0.smithi129.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-03-25T23:03:04.585 INFO:tasks.workunit.client.0.smithi129.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 � u � 205 \0 \0 2024-03-25T23:03:04.585 INFO:tasks.workunit.client.0.smithi129.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-03-25T23:03:04.585 INFO:tasks.workunit.client.0.smithi129.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-03-25T23:03:04.585 INFO:tasks.workunit.client.0.smithi129.stdout:0000073 2024-03-25T23:03:04.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-03-25T23:03:04.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-25T23:03:06.404 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.RNQnos9gJ5_the_val 2024-03-25T23:03:07.723 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-25T23:03:07.723 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:129: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 ' 2024-03-25T23:03:07.724 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: echo --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 2024-03-25T23:03:07.724 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-03-25T23:03:07.724 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-25T23:03:07.724 INFO:tasks.workunit.client.0.smithi129.stdout:Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/ 2024-03-25T23:03:07.724 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: echo 'Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/' 2024-03-25T23:03:07.725 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: expr 3 - 1 2024-03-25T23:03:07.726 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: seq 0 2 2024-03-25T23:03:07.728 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-25T23:03:07.728 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=99b4bc82-c34e-4895-95ea-2717aa85e928 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-03-25T23:03:07.728 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 0 2024-03-25T23:03:07.728 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-03-25T23:03:07.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:03:07.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-25T23:03:07.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:03:07.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/0 2024-03-25T23:03:07.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=99b4bc82-c34e-4895-95ea-2717aa85e928 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-03-25T23:03:07.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:03:07.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:03:07.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:03:07.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/0' 2024-03-25T23:03:07.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/0/journal' 2024-03-25T23:03:07.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:03:07.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:03:07.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-03-25T23:03:07.730 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:03:07.730 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:03:07.730 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:03:07.731 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:03:07.731 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:03:07.731 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20201 2024-03-25T23:03:07.731 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20201/$cluster-$name.asok' 2024-03-25T23:03:07.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20201/$cluster-$name.asok' 2024-03-25T23:03:07.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:03:07.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-03-25T23:03:07.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-03-25T23:03:07.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:03:07.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:03:07.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:03:07.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:03:07.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:03:07.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:03:07.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/0 2024-03-25T23:03:07.732 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.0 2024-03-25T23:03:07.733 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-25T23:03:07.733 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=99b4bc82-c34e-4895-95ea-2717aa85e928 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/0 --osd-journal=td/osd-mapper/0/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.20201/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:03:07.733 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/0/whoami 2024-03-25T23:03:07.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-25T23:03:07.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:03:07.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:03:07.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:03:07.782 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:07.781+0000 7f519e5a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:03:07.785 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:07.785+0000 7f519e5a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:03:07.787 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:07.785+0000 7f519e5a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:03:08.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-25T23:03:08.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:03:08.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T23:03:08.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:03:08.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:03:08.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:03:08.063 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:03:08.063 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:03:08.063 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:03:08.063 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:03:08.315 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:08.313+0000 7f519e5a27c0 -1 Falling back to public interface 2024-03-25T23:03:08.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:03:09.396 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:03:09.396 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:03:09.396 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:03:09.397 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:03:09.397 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:03:09.397 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:03:09.721 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:03:10.726 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:03:10.726 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:03:10.726 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:03:10.727 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:03:10.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:03:10.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:03:10.818 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:10.817+0000 7f519e5a27c0 -1 osd.0 30 log_to_monitors true 2024-03-25T23:03:11.117 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:03:12.118 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:03:12.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:03:12.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:03:12.119 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:03:12.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:03:12.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:03:12.454 INFO:tasks.workunit.client.0.smithi129.stdout:osd.0 up in weight 1 up_from 33 up_thru 33 down_at 31 last_clean_interval [5,30) [v2:127.0.0.1:6800/4255486980,v1:127.0.0.1:6801/4255486980] [v2:127.0.0.1:6802/4255486980,v1:127.0.0.1:6803/4255486980] exists,up d0f8b0f8-175f-43ff-920c-011f8bc0c712 2024-03-25T23:03:12.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:03:12.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:03:12.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:03:12.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-25T23:03:12.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=99b4bc82-c34e-4895-95ea-2717aa85e928 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-03-25T23:03:12.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 1 2024-03-25T23:03:12.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-03-25T23:03:12.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:03:12.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-25T23:03:12.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:03:12.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/1 2024-03-25T23:03:12.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=99b4bc82-c34e-4895-95ea-2717aa85e928 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-03-25T23:03:12.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:03:12.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:03:12.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:03:12.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/1' 2024-03-25T23:03:12.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/1/journal' 2024-03-25T23:03:12.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:03:12.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:03:12.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-03-25T23:03:12.458 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:03:12.458 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:03:12.458 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:03:12.459 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:03:12.459 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:03:12.459 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20201 2024-03-25T23:03:12.459 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20201/$cluster-$name.asok' 2024-03-25T23:03:12.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20201/$cluster-$name.asok' 2024-03-25T23:03:12.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:03:12.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-03-25T23:03:12.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-03-25T23:03:12.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:03:12.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:03:12.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:03:12.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:03:12.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:03:12.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:03:12.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/1 2024-03-25T23:03:12.460 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.1 2024-03-25T23:03:12.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-25T23:03:12.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=99b4bc82-c34e-4895-95ea-2717aa85e928 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/1 --osd-journal=td/osd-mapper/1/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.20201/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:03:12.461 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/1/whoami 2024-03-25T23:03:12.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-25T23:03:12.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:03:12.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:03:12.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:03:12.497 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:12.497+0000 7f0d03f217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:03:12.508 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:12.505+0000 7f0d03f217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:03:12.509 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:12.509+0000 7f0d03f217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:03:12.795 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-25T23:03:12.795 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:03:12.795 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T23:03:12.796 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:03:12.796 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:03:12.796 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:03:12.796 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:03:12.796 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:03:12.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:03:12.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:03:13.059 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:13.057+0000 7f0d03f217c0 -1 Falling back to public interface 2024-03-25T23:03:13.143 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:03:14.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:03:14.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:03:14.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:03:14.145 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:03:14.146 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:03:14.146 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:03:14.462 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:03:15.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:03:15.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:03:15.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:03:15.464 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:03:15.465 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:03:15.465 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:03:15.534 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:15.533+0000 7f0d03f217c0 -1 osd.1 30 log_to_monitors true 2024-03-25T23:03:15.813 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:03:16.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:03:16.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:03:16.814 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:03:16.815 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:03:16.815 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:03:16.815 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:03:17.192 INFO:tasks.workunit.client.0.smithi129.stdout:osd.1 up in weight 1 up_from 36 up_thru 36 down_at 31 last_clean_interval [10,30) [v2:127.0.0.1:6810/1426652748,v1:127.0.0.1:6811/1426652748] [v2:127.0.0.1:6812/1426652748,v1:127.0.0.1:6813/1426652748] exists,up 7ef91d1c-2bf1-4f0c-86db-44d843c5755b 2024-03-25T23:03:17.196 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:03:17.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:03:17.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:03:17.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-25T23:03:17.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=99b4bc82-c34e-4895-95ea-2717aa85e928 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-03-25T23:03:17.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 2 2024-03-25T23:03:17.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-03-25T23:03:17.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:03:17.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-25T23:03:17.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:03:17.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/2 2024-03-25T23:03:17.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=99b4bc82-c34e-4895-95ea-2717aa85e928 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-03-25T23:03:17.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:03:17.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:03:17.198 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:03:17.198 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/2' 2024-03-25T23:03:17.198 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/2/journal' 2024-03-25T23:03:17.198 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:03:17.198 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:03:17.198 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-03-25T23:03:17.198 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:03:17.198 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:03:17.198 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:03:17.198 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:03:17.198 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:03:17.198 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20201 2024-03-25T23:03:17.198 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20201/$cluster-$name.asok' 2024-03-25T23:03:17.198 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20201/$cluster-$name.asok' 2024-03-25T23:03:17.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:03:17.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-03-25T23:03:17.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-03-25T23:03:17.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:03:17.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:03:17.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:03:17.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:03:17.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:03:17.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:03:17.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/2 2024-03-25T23:03:17.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-25T23:03:17.200 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.2 2024-03-25T23:03:17.200 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=99b4bc82-c34e-4895-95ea-2717aa85e928 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/2 --osd-journal=td/osd-mapper/2/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.20201/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:03:17.200 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/2/whoami 2024-03-25T23:03:17.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-25T23:03:17.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:03:17.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:03:17.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:03:17.239 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:17.237+0000 7f218c2627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:03:17.264 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:17.261+0000 7f218c2627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:03:17.265 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:17.265+0000 7f218c2627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:03:17.571 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-25T23:03:17.571 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:03:17.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-25T23:03:17.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:03:17.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:03:17.572 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:03:17.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:03:17.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:03:17.573 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:03:17.573 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:03:17.807 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:17.805+0000 7f218c2627c0 -1 Falling back to public interface 2024-03-25T23:03:17.906 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:03:18.906 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:03:18.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:03:18.907 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:03:18.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:03:18.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:03:18.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:03:19.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:03:20.241 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:03:20.241 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:03:20.242 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:03:20.242 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:03:20.242 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:03:20.242 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:03:20.363 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:20.361+0000 7f218c2627c0 -1 osd.2 30 log_to_monitors true 2024-03-25T23:03:20.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:03:21.574 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:03:21.574 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:03:21.574 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:03:21.574 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:03:21.575 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:03:21.575 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:03:22.025 INFO:tasks.workunit.client.0.smithi129.stdout:osd.2 up in weight 1 up_from 40 up_thru 40 down_at 31 last_clean_interval [15,30) [v2:127.0.0.1:6818/1796608602,v1:127.0.0.1:6819/1796608602] [v2:127.0.0.1:6820/1796608602,v1:127.0.0.1:6821/1796608602] exists,up 6a2a0a84-9b88-4dbf-ae30-3c6f603fd676 2024-03-25T23:03:22.025 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:03:22.025 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:03:22.026 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:03:22.026 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:136: TEST_truncated_sna_record: sleep 1 2024-03-25T23:03:23.028 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: expr 3 - 1 2024-03-25T23:03:23.029 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: seq 0 2 2024-03-25T23:03:23.030 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-25T23:03:23.031 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.0 version 2024-03-25T23:03:23.171 INFO:tasks.workunit.client.0.smithi129.stdout:{ 2024-03-25T23:03:23.171 INFO:tasks.workunit.client.0.smithi129.stdout: "version": "19.0.0-2416-g4354a201", 2024-03-25T23:03:23.171 INFO:tasks.workunit.client.0.smithi129.stdout: "release": "squid", 2024-03-25T23:03:23.171 INFO:tasks.workunit.client.0.smithi129.stdout: "release_type": "dev" 2024-03-25T23:03:23.171 INFO:tasks.workunit.client.0.smithi129.stdout:} 2024-03-25T23:03:23.185 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-25T23:03:23.185 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.1 version 2024-03-25T23:03:23.318 INFO:tasks.workunit.client.0.smithi129.stdout:{ 2024-03-25T23:03:23.318 INFO:tasks.workunit.client.0.smithi129.stdout: "version": "19.0.0-2416-g4354a201", 2024-03-25T23:03:23.318 INFO:tasks.workunit.client.0.smithi129.stdout: "release": "squid", 2024-03-25T23:03:23.318 INFO:tasks.workunit.client.0.smithi129.stdout: "release_type": "dev" 2024-03-25T23:03:23.318 INFO:tasks.workunit.client.0.smithi129.stdout:} 2024-03-25T23:03:23.333 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-25T23:03:23.333 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.2 version 2024-03-25T23:03:23.466 INFO:tasks.workunit.client.0.smithi129.stdout:{ 2024-03-25T23:03:23.466 INFO:tasks.workunit.client.0.smithi129.stdout: "version": "19.0.0-2416-g4354a201", 2024-03-25T23:03:23.467 INFO:tasks.workunit.client.0.smithi129.stdout: "release": "squid", 2024-03-25T23:03:23.467 INFO:tasks.workunit.client.0.smithi129.stdout: "release_type": "dev" 2024-03-25T23:03:23.467 INFO:tasks.workunit.client.0.smithi129.stdout:} 2024-03-25T23:03:23.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:142: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2024-03-25T23:03:23.510 INFO:tasks.workunit.client.0.smithi129.stdout:{ 2024-03-25T23:03:23.510 INFO:tasks.workunit.client.0.smithi129.stdout: "name": "objxxx", 2024-03-25T23:03:23.510 INFO:tasks.workunit.client.0.smithi129.stdout: "clones": [ 2024-03-25T23:03:23.510 INFO:tasks.workunit.client.0.smithi129.stdout: { 2024-03-25T23:03:23.510 INFO:tasks.workunit.client.0.smithi129.stdout: "id": 2, 2024-03-25T23:03:23.510 INFO:tasks.workunit.client.0.smithi129.stdout: "snapshots": [ 2024-03-25T23:03:23.510 INFO:tasks.workunit.client.0.smithi129.stdout: { 2024-03-25T23:03:23.511 INFO:tasks.workunit.client.0.smithi129.stdout: "id": 1, 2024-03-25T23:03:23.511 INFO:tasks.workunit.client.0.smithi129.stdout: "name": "snap01" 2024-03-25T23:03:23.511 INFO:tasks.workunit.client.0.smithi129.stdout: }, 2024-03-25T23:03:23.511 INFO:tasks.workunit.client.0.smithi129.stdout: { 2024-03-25T23:03:23.511 INFO:tasks.workunit.client.0.smithi129.stdout: "id": 2, 2024-03-25T23:03:23.511 INFO:tasks.workunit.client.0.smithi129.stdout: "name": "snap02" 2024-03-25T23:03:23.511 INFO:tasks.workunit.client.0.smithi129.stdout: } 2024-03-25T23:03:23.511 INFO:tasks.workunit.client.0.smithi129.stdout: ], 2024-03-25T23:03:23.511 INFO:tasks.workunit.client.0.smithi129.stdout: "size": 6, 2024-03-25T23:03:23.511 INFO:tasks.workunit.client.0.smithi129.stdout: "overlaps": [] 2024-03-25T23:03:23.511 INFO:tasks.workunit.client.0.smithi129.stdout: }, 2024-03-25T23:03:23.511 INFO:tasks.workunit.client.0.smithi129.stdout: { 2024-03-25T23:03:23.511 INFO:tasks.workunit.client.0.smithi129.stdout: "id": 3, 2024-03-25T23:03:23.511 INFO:tasks.workunit.client.0.smithi129.stdout: "snapshots": [ 2024-03-25T23:03:23.511 INFO:tasks.workunit.client.0.smithi129.stdout: { 2024-03-25T23:03:23.512 INFO:tasks.workunit.client.0.smithi129.stdout: "id": 3, 2024-03-25T23:03:23.512 INFO:tasks.workunit.client.0.smithi129.stdout: "name": "snap13" 2024-03-25T23:03:23.512 INFO:tasks.workunit.client.0.smithi129.stdout: } 2024-03-25T23:03:23.512 INFO:tasks.workunit.client.0.smithi129.stdout: ], 2024-03-25T23:03:23.512 INFO:tasks.workunit.client.0.smithi129.stdout: "size": 4, 2024-03-25T23:03:23.512 INFO:tasks.workunit.client.0.smithi129.stdout: "overlaps": [] 2024-03-25T23:03:23.513 INFO:tasks.workunit.client.0.smithi129.stdout: }, 2024-03-25T23:03:23.513 INFO:tasks.workunit.client.0.smithi129.stdout: { 2024-03-25T23:03:23.513 INFO:tasks.workunit.client.0.smithi129.stdout: "id": 5, 2024-03-25T23:03:23.513 INFO:tasks.workunit.client.0.smithi129.stdout: "snapshots": [ 2024-03-25T23:03:23.513 INFO:tasks.workunit.client.0.smithi129.stdout: { 2024-03-25T23:03:23.513 INFO:tasks.workunit.client.0.smithi129.stdout: "id": 4, 2024-03-25T23:03:23.513 INFO:tasks.workunit.client.0.smithi129.stdout: "name": "snap24" 2024-03-25T23:03:23.513 INFO:tasks.workunit.client.0.smithi129.stdout: }, 2024-03-25T23:03:23.513 INFO:tasks.workunit.client.0.smithi129.stdout: { 2024-03-25T23:03:23.513 INFO:tasks.workunit.client.0.smithi129.stdout: "id": 5, 2024-03-25T23:03:23.513 INFO:tasks.workunit.client.0.smithi129.stdout: "name": "snap25" 2024-03-25T23:03:23.513 INFO:tasks.workunit.client.0.smithi129.stdout: } 2024-03-25T23:03:23.513 INFO:tasks.workunit.client.0.smithi129.stdout: ], 2024-03-25T23:03:23.513 INFO:tasks.workunit.client.0.smithi129.stdout: "size": 6, 2024-03-25T23:03:23.513 INFO:tasks.workunit.client.0.smithi129.stdout: "overlaps": [] 2024-03-25T23:03:23.513 INFO:tasks.workunit.client.0.smithi129.stdout: }, 2024-03-25T23:03:23.513 INFO:tasks.workunit.client.0.smithi129.stdout: { 2024-03-25T23:03:23.514 INFO:tasks.workunit.client.0.smithi129.stdout: "id": "head", 2024-03-25T23:03:23.514 INFO:tasks.workunit.client.0.smithi129.stdout: "snapshots": [], 2024-03-25T23:03:23.514 INFO:tasks.workunit.client.0.smithi129.stdout: "size": 6 2024-03-25T23:03:23.514 INFO:tasks.workunit.client.0.smithi129.stdout: } 2024-03-25T23:03:23.514 INFO:tasks.workunit.client.0.smithi129.stdout: ] 2024-03-25T23:03:23.514 INFO:tasks.workunit.client.0.smithi129.stdout:} 2024-03-25T23:03:23.514 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:145: TEST_truncated_sna_record: ceph osd unset nodeep-scrub 2024-03-25T23:03:23.898 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is unset 2024-03-25T23:03:23.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:146: TEST_truncated_sna_record: ceph osd unset noscrub 2024-03-25T23:03:24.325 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is unset 2024-03-25T23:03:24.345 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-03-25T23:03:24.345 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: jq -r '.up[0]' 2024-03-25T23:03:24.673 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: local cur_prim=1 2024-03-25T23:03:24.673 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:150: TEST_truncated_sna_record: ceph pg dump pgs 2024-03-25T23:03:24.945 INFO:tasks.workunit.client.0.smithi129.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-25T23:03:24.945 INFO:tasks.workunit.client.0.smithi129.stdout:1.3 4 0 4 0 0 22 0 0 7 0 7 active+undersized+degraded 2024-03-25T23:03:16.646662+0000 28'7 38:58 [1,0] 1 [1,0] 1 28'7 2024-03-25T23:02:38.880300+0000 28'7 2024-03-25T23:02:38.880300+0000 0 1 periodic scrub scheduled @ 2024-03-26T23:02:38.880300+0000 4 0 2024-03-25T23:03:24.945 INFO:tasks.workunit.client.0.smithi129.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 peering 2024-03-25T23:03:20.926020+0000 0'0 39:30 [0,1,2] 0 [0,1,2] 0 0'0 2024-03-25T23:02:28.045695+0000 0'0 2024-03-25T23:02:28.045695+0000 0 0 no scrub is scheduled 0 0 2024-03-25T23:03:24.945 INFO:tasks.workunit.client.0.smithi129.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 undersized+peered 2024-03-25T23:03:11.807481+0000 0'0 35:10 [0] 0 [0] 0 0'0 2024-03-25T23:02:28.045695+0000 0'0 2024-03-25T23:02:28.045695+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:02:28.045695+0000 0 0 2024-03-25T23:03:24.945 INFO:tasks.workunit.client.0.smithi129.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-03-25T23:03:16.646910+0000 0'0 38:22 [1,0] 1 [1,0] 1 0'0 2024-03-25T23:02:28.045695+0000 0'0 2024-03-25T23:02:28.045695+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:02:28.045695+0000 0 0 2024-03-25T23:03:24.945 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:03:24.946 INFO:tasks.workunit.client.0.smithi129.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-25T23:03:24.946 INFO:tasks.workunit.client.0.smithi129.stderr:dumped pgs 2024-03-25T23:03:24.960 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:151: TEST_truncated_sna_record: sleep 2 2024-03-25T23:03:26.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:152: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-03-25T23:03:27.101 INFO:tasks.workunit.client.0.smithi129.stdout:{ 2024-03-25T23:03:27.101 INFO:tasks.workunit.client.0.smithi129.stdout: "deep": true, 2024-03-25T23:03:27.101 INFO:tasks.workunit.client.0.smithi129.stdout: "must": true, 2024-03-25T23:03:27.101 INFO:tasks.workunit.client.0.smithi129.stdout: "stamp": "0.000000" 2024-03-25T23:03:27.101 INFO:tasks.workunit.client.0.smithi129.stdout:} 2024-03-25T23:03:27.115 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:153: TEST_truncated_sna_record: sleep 5 2024-03-25T23:03:32.117 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:154: TEST_truncated_sna_record: ceph pg dump pgs 2024-03-25T23:03:32.408 INFO:tasks.workunit.client.0.smithi129.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-25T23:03:32.408 INFO:tasks.workunit.client.0.smithi129.stdout:1.3 4 0 0 0 0 22 0 0 7 0 7 active+clean 2024-03-25T23:03:27.532138+0000 28'7 44:84 [1,2,0] 1 [1,2,0] 1 28'7 2024-03-25T23:03:27.531934+0000 28'7 2024-03-25T23:03:27.531934+0000 0 1 periodic scrub scheduled @ 2024-03-26T23:03:27.531934+0000 4 0 2024-03-25T23:03:32.408 INFO:tasks.workunit.client.0.smithi129.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T23:03:21.697311+0000 0'0 44:45 [0,1,2] 0 [0,1,2] 0 0'0 2024-03-25T23:02:28.045695+0000 0'0 2024-03-25T23:02:28.045695+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:02:28.045695+0000 0 0 2024-03-25T23:03:32.408 INFO:tasks.workunit.client.0.smithi129.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T23:03:21.779387+0000 0'0 44:29 [2,0,1] 2 [2,0,1] 2 0'0 2024-03-25T23:02:28.045695+0000 0'0 2024-03-25T23:02:28.045695+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:02:28.045695+0000 0 0 2024-03-25T23:03:32.408 INFO:tasks.workunit.client.0.smithi129.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T23:03:22.217431+0000 0'0 44:38 [1,0,2] 1 [1,0,2] 1 0'0 2024-03-25T23:02:28.045695+0000 0'0 2024-03-25T23:02:28.045695+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:02:28.045695+0000 0 0 2024-03-25T23:03:32.408 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:03:32.408 INFO:tasks.workunit.client.0.smithi129.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-25T23:03:32.408 INFO:tasks.workunit.client.0.smithi129.stderr:dumped pgs 2024-03-25T23:03:32.423 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-25T23:03:32.423 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-03-25T23:03:32.426 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:03:27.529+0000 7f0ce66a8640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2024-03-25T23:03:32.426 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:156: TEST_truncated_sna_record: grep -a -q ERR td/osd-mapper/osd.1.log 2024-03-25T23:03:32.429 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-03-25T23:03:32.429 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: wc -l 2024-03-25T23:03:32.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: local prev_err_cnt=1 2024-03-25T23:03:32.432 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:161: TEST_truncated_sna_record: echo 'prev count: 1' 2024-03-25T23:03:32.432 INFO:tasks.workunit.client.0.smithi129.stdout:prev count: 1 2024-03-25T23:03:32.432 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:164: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-03-25T23:03:32.547 INFO:tasks.workunit.client.0.smithi129.stdout:{ 2024-03-25T23:03:32.547 INFO:tasks.workunit.client.0.smithi129.stdout: "deep": true, 2024-03-25T23:03:32.547 INFO:tasks.workunit.client.0.smithi129.stdout: "must": true, 2024-03-25T23:03:32.547 INFO:tasks.workunit.client.0.smithi129.stdout: "stamp": "0.000000" 2024-03-25T23:03:32.547 INFO:tasks.workunit.client.0.smithi129.stdout:} 2024-03-25T23:03:32.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:165: TEST_truncated_sna_record: sleep 5 2024-03-25T23:03:37.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:166: TEST_truncated_sna_record: ceph pg dump pgs 2024-03-25T23:03:37.857 INFO:tasks.workunit.client.0.smithi129.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-25T23:03:37.857 INFO:tasks.workunit.client.0.smithi129.stdout:1.3 4 0 0 0 0 22 0 0 7 0 7 active+clean 2024-03-25T23:03:33.513234+0000 28'7 44:93 [1,2,0] 1 [1,2,0] 1 28'7 2024-03-25T23:03:33.513015+0000 28'7 2024-03-25T23:03:33.513015+0000 0 1 periodic scrub scheduled @ 2024-03-26T23:03:33.513015+0000 4 0 2024-03-25T23:03:37.858 INFO:tasks.workunit.client.0.smithi129.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T23:03:21.697311+0000 0'0 44:45 [0,1,2] 0 [0,1,2] 0 0'0 2024-03-25T23:02:28.045695+0000 0'0 2024-03-25T23:02:28.045695+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:02:28.045695+0000 0 0 2024-03-25T23:03:37.858 INFO:tasks.workunit.client.0.smithi129.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T23:03:21.779387+0000 0'0 44:29 [2,0,1] 2 [2,0,1] 2 0'0 2024-03-25T23:02:28.045695+0000 0'0 2024-03-25T23:02:28.045695+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:02:28.045695+0000 0 0 2024-03-25T23:03:37.858 INFO:tasks.workunit.client.0.smithi129.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T23:03:22.217431+0000 0'0 44:38 [1,0,2] 1 [1,0,2] 1 0'0 2024-03-25T23:02:28.045695+0000 0'0 2024-03-25T23:02:28.045695+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:02:28.045695+0000 0 0 2024-03-25T23:03:37.858 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:03:37.858 INFO:tasks.workunit.client.0.smithi129.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-25T23:03:37.858 INFO:tasks.workunit.client.0.smithi129.stderr:dumped pgs 2024-03-25T23:03:37.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-25T23:03:37.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-03-25T23:03:37.877 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:03:27.529+0000 7f0ce66a8640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2024-03-25T23:03:37.878 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: wc -l 2024-03-25T23:03:37.878 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-03-25T23:03:37.881 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: local current_err_cnt=1 2024-03-25T23:03:37.881 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-25T23:03:37.881 INFO:tasks.workunit.client.0.smithi129.stdout:current count: 1 2024-03-25T23:03:37.882 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: echo 'current count: 1' 2024-03-25T23:03:37.882 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:170: TEST_truncated_sna_record: (( current_err_cnt == prev_err_cnt )) 2024-03-25T23:03:37.882 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:171: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2024-03-25T23:03:37.882 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:03:37.882 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:03:37.882 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:03:37.882 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:03:37.882 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:03:38.190 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:03:38.191 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:172: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2024-03-25T23:03:38.191 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-25T23:03:38.191 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-03-25T23:03:38.191 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_' 2024-03-25T23:03:38.191 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:174: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-03-25T23:03:39.018 INFO:tasks.workunit.client.0.smithi129.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-03-25T23:03:39.019 INFO:tasks.workunit.client.0.smithi129.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-03-25T23:03:39.019 INFO:tasks.workunit.client.0.smithi129.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-03-25T23:03:39.019 INFO:tasks.workunit.client.0.smithi129.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-25T23:03:39.019 INFO:tasks.workunit.client.0.smithi129.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-03-25T23:03:39.019 INFO:tasks.workunit.client.0.smithi129.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-03-25T23:03:39.020 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-03-25T23:03:39.020 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2024-03-25T23:03:39.020 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-03-25T23:03:39.020 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: wc -l 2024-03-25T23:03:39.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: local num_sna_full=5 2024-03-25T23:03:39.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:177: TEST_truncated_sna_record: (( num_sna_full == num_sna_b4 )) 2024-03-25T23:03:39.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:178: TEST_truncated_sna_record: return 0 2024-03-25T23:03:39.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:24: run: teardown td/osd-mapper 2024-03-25T23:03:39.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-03-25T23:03:39.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T23:03:39.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-03-25T23:03:39.837 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:03:39.837 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:03:39.837 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:03:39.837 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:03:39.837 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:03:39.947 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:03:39.947 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T23:03:39.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T23:03:39.949 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T23:03:39.950 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T23:03:39.950 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T23:03:39.950 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T23:03:39.951 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:03:39.951 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T23:03:39.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T23:03:39.952 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:03:39.953 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T23:03:39.954 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T23:03:39.954 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-03-25T23:03:39.971 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T23:03:39.971 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:03:39.971 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20201 2024-03-25T23:03:39.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20201 2024-03-25T23:03:39.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T23:03:39.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T23:03:39.974 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-03-25T23:03:39.974 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-mapper 0 2024-03-25T23:03:39.974 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-03-25T23:03:39.974 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-03-25T23:03:39.974 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-03-25T23:03:39.974 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:03:39.974 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:03:39.974 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:03:39.974 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:03:39.974 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:03:39.976 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:03:39.976 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T23:03:39.976 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T23:03:39.977 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T23:03:39.978 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T23:03:39.978 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T23:03:39.978 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T23:03:39.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:03:39.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T23:03:39.980 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T23:03:39.980 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:03:39.980 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T23:03:39.982 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-03-25T23:03:39.982 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-03-25T23:03:39.983 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T23:03:39.983 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:03:39.983 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20201 2024-03-25T23:03:39.983 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20201 2024-03-25T23:03:39.985 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T23:03:39.985 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T23:03:39.985 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-03-25T23:03:39.985 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-25T23:03:39.986 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-25T23:03:40.041 INFO:tasks.workunit:Running workunit scrub/osd-recovery-scrub.sh... 2024-03-25T23:03:40.041 DEBUG:teuthology.orchestra.run.smithi129:workunit test scrub/osd-recovery-scrub.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4354a20109fd25703f6611a1a8f196608a256149 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh 2024-03-25T23:03:40.099 INFO:tasks.workunit.client.0.smithi129.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-03-25T23:03:40.105 INFO:tasks.workunit.client.0.smithi129.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-03-25T23:03:40.105 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-03-25T23:03:40.105 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-03-25T23:03:40.105 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-03-25T23:03:40.105 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-03-25T23:03:40.105 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-03-25T23:03:40.105 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-03-25T23:03:40.105 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-03-25T23:03:40.105 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-03-25T23:03:40.105 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-recovery-scrub 2024-03-25T23:03:40.105 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:20: run: local dir=td/osd-recovery-scrub 2024-03-25T23:03:40.106 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:21: run: shift 2024-03-25T23:03:40.106 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: export CEPH_MON=127.0.0.1:7124 2024-03-25T23:03:40.106 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: CEPH_MON=127.0.0.1:7124 2024-03-25T23:03:40.106 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:24: run: export CEPH_ARGS 2024-03-25T23:03:40.106 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: uuidgen 2024-03-25T23:03:40.107 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: CEPH_ARGS+='--fsid=4de2c693-e629-4160-940c-ecd79a0b43d9 --auth-supported=none ' 2024-03-25T23:03:40.107 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2024-03-25T23:03:40.107 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:27: run: CEPH_ARGS+='--osd-op-queue=wpq ' 2024-03-25T23:03:40.107 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:29: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-03-25T23:03:40.107 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: set 2024-03-25T23:03:40.107 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-03-25T23:03:40.109 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: local 'funcs=TEST_recovery_scrub_1 2024-03-25T23:03:40.109 INFO:tasks.workunit.client.0.smithi129.stderr:TEST_recovery_scrub_2' 2024-03-25T23:03:40.109 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: for func in $funcs 2024-03-25T23:03:40.110 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: TEST_recovery_scrub_1 td/osd-recovery-scrub 2024-03-25T23:03:40.110 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:39: TEST_recovery_scrub_1: local dir=td/osd-recovery-scrub 2024-03-25T23:03:40.110 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:40: TEST_recovery_scrub_1: local poolname=test 2024-03-25T23:03:40.110 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:42: TEST_recovery_scrub_1: TESTDATA=testdata.36561 2024-03-25T23:03:40.110 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:43: TEST_recovery_scrub_1: OSDS=4 2024-03-25T23:03:40.110 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:44: TEST_recovery_scrub_1: PGS=1 2024-03-25T23:03:40.110 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:45: TEST_recovery_scrub_1: OBJECTS=100 2024-03-25T23:03:40.110 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:46: TEST_recovery_scrub_1: ERRORS=0 2024-03-25T23:03:40.110 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:48: TEST_recovery_scrub_1: setup td/osd-recovery-scrub 2024-03-25T23:03:40.110 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-scrub 2024-03-25T23:03:40.110 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-scrub 2024-03-25T23:03:40.110 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-03-25T23:03:40.110 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T23:03:40.111 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-03-25T23:03:40.111 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:03:40.111 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:03:40.111 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:03:40.111 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:03:40.111 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:03:40.111 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:03:40.111 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T23:03:40.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T23:03:40.112 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T23:03:40.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T23:03:40.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T23:03:40.114 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T23:03:40.115 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:03:40.115 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T23:03:40.115 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T23:03:40.115 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:03:40.116 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T23:03:40.117 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T23:03:40.117 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-03-25T23:03:40.119 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T23:03:40.119 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:03:40.119 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36561 2024-03-25T23:03:40.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36561 2024-03-25T23:03:40.120 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T23:03:40.120 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T23:03:40.120 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-scrub 2024-03-25T23:03:40.121 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-25T23:03:40.122 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:03:40.122 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36561 2024-03-25T23:03:40.122 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.36561 2024-03-25T23:03:40.124 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-25T23:03:40.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-03-25T23:03:40.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-03-25T23:03:40.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-25T23:03:40.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2024-03-25T23:03:40.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:49: TEST_recovery_scrub_1: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-03-25T23:03:40.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-scrub 2024-03-25T23:03:40.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-25T23:03:40.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-25T23:03:40.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-25T23:03:40.125 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-scrub/a 2024-03-25T23:03:40.125 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-03-25T23:03:40.625 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-25T23:03:40.625 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:03:40.626 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:03:40.626 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:03:40.626 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:03:40.626 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36561 2024-03-25T23:03:40.626 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36561/$cluster-$name.asok' 2024-03-25T23:03:40.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.36561/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-03-25T23:03:40.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-25T23:03:40.835 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-25T23:03:40.835 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T23:03:40.835 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T23:03:40.835 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-25T23:03:40.836 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-25T23:03:40.836 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T23:03:40.836 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T23:03:40.837 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T23:03:40.837 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T23:03:40.837 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:03:40.837 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36561 2024-03-25T23:03:40.837 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36561/ceph-mon.a.asok 2024-03-25T23:03:40.837 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T23:03:40.837 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.36561/ceph-mon.a.asok config get fsid 2024-03-25T23:03:40.956 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-25T23:03:40.956 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T23:03:40.957 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T23:03:40.957 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-25T23:03:40.958 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T23:03:40.958 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-25T23:03:40.958 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T23:03:40.958 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T23:03:40.958 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T23:03:40.958 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:03:40.958 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36561 2024-03-25T23:03:40.958 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36561/ceph-mon.a.asok 2024-03-25T23:03:40.959 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T23:03:40.959 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.36561/ceph-mon.a.asok config get mon_host 2024-03-25T23:03:41.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:50: TEST_recovery_scrub_1: run_mgr td/osd-recovery-scrub x 2024-03-25T23:03:41.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-scrub 2024-03-25T23:03:41.068 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-25T23:03:41.068 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-25T23:03:41.068 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-25T23:03:41.068 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-scrub/x 2024-03-25T23:03:41.068 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-25T23:03:41.952 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-25T23:03:41.953 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:03:41.953 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:03:41.953 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:03:41.953 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:03:41.953 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36561 2024-03-25T23:03:41.953 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36561/$cluster-$name.asok' 2024-03-25T23:03:41.954 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T23:03:41.955 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.36561/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T23:03:41.990 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 ' 2024-03-25T23:03:41.990 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:52: TEST_recovery_scrub_1: ceph_osd_args+='--osd_scrub_backoff_ratio=0 ' 2024-03-25T23:03:41.990 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:53: TEST_recovery_scrub_1: ceph_osd_args+='--osd_stats_update_period_not_scrubbing=3 ' 2024-03-25T23:03:41.990 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:54: TEST_recovery_scrub_1: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2024-03-25T23:03:41.990 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: expr 4 - 1 2024-03-25T23:03:41.991 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: seq 0 3 2024-03-25T23:03:41.994 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-25T23:03:41.994 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=false 2024-03-25T23:03:41.995 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-25T23:03:41.995 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:03:41.995 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-25T23:03:41.995 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:03:41.995 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/0 2024-03-25T23:03:41.995 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4de2c693-e629-4160-940c-ecd79a0b43d9 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-25T23:03:41.995 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:03:41.995 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:03:41.995 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:03:41.995 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2024-03-25T23:03:41.995 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2024-03-25T23:03:41.995 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:03:41.995 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:03:41.995 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-25T23:03:41.996 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:03:41.996 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:03:41.996 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:03:41.996 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:03:41.996 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:03:41.996 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36561 2024-03-25T23:03:41.996 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36561/$cluster-$name.asok' 2024-03-25T23:03:41.996 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36561/$cluster-$name.asok' 2024-03-25T23:03:41.996 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:03:41.996 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:03:41.996 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:03:41.996 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-25T23:03:41.996 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-25T23:03:41.996 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:03:41.996 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:03:41.997 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:03:41.997 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:03:41.997 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:03:41.997 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-03-25T23:03:41.997 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/0 2024-03-25T23:03:41.997 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:03:41.997 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fbcb1bb8-89a6-4108-b8af-81c759b5546a 2024-03-25T23:03:41.998 INFO:tasks.workunit.client.0.smithi129.stdout:add osd0 fbcb1bb8-89a6-4108-b8af-81c759b5546a 2024-03-25T23:03:41.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 fbcb1bb8-89a6-4108-b8af-81c759b5546a' 2024-03-25T23:03:41.998 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:03:42.013 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDOAgJmx9XPABAApSPfRIyAkFKmZzbqwb4iHQ== 2024-03-25T23:03:42.014 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDOAgJmx9XPABAApSPfRIyAkFKmZzbqwb4iHQ=="}' 2024-03-25T23:03:42.014 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fbcb1bb8-89a6-4108-b8af-81c759b5546a -i td/osd-recovery-scrub/0/new.json 2024-03-25T23:03:42.314 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:03:42.333 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/0/new.json 2024-03-25T23:03:42.334 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=4de2c693-e629-4160-940c-ecd79a0b43d9 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36561/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQDOAgJmx9XPABAApSPfRIyAkFKmZzbqwb4iHQ== --osd-uuid fbcb1bb8-89a6-4108-b8af-81c759b5546a 2024-03-25T23:03:42.369 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:42.368+0000 7fd41b0197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:03:42.429 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:42.380+0000 7fd41b0197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:03:42.429 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:42.380+0000 7fd41b0197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:03:42.429 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:42.380+0000 7fd41b0197c0 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-03-25T23:03:42.429 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:42.380+0000 7fd41b0197c0 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-03-25T23:03:42.430 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:42.380+0000 7fd41b0197c0 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-03-25T23:03:42.482 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:42.480+0000 7fd41b0197c0 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2024-03-25T23:03:48.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2024-03-25T23:03:48.931 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:03:48.933 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd0 key to auth repository 2024-03-25T23:03:48.933 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-25T23:03:48.933 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:03:49.325 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.0 2024-03-25T23:03:49.325 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-25T23:03:49.325 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:03:49.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:03:49.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=4de2c693-e629-4160-940c-ecd79a0b43d9 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36561/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-03-25T23:03:49.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:03:49.357 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:49.356+0000 7f76aaaff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:03:49.383 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:49.380+0000 7f76aaaff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:03:49.384 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:49.380+0000 7f76aaaff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:03:49.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-25T23:03:49.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:03:49.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T23:03:49.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:03:49.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:03:49.659 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:03:49.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:03:49.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:03:49.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:03:49.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:03:49.940 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:49.936+0000 7f76aaaff7c0 -1 Falling back to public interface 2024-03-25T23:03:50.002 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:03:51.003 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:03:51.004 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:03:51.004 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:03:51.004 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:03:51.005 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:03:51.005 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:03:51.323 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:03:52.324 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:03:52.324 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:03:52.325 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:03:52.325 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:03:52.325 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:03:52.325 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:03:52.575 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:52.572+0000 7f76aaaff7c0 -1 osd.0 0 log_to_monitors true 2024-03-25T23:03:52.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:03:53.629 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:03:53.629 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:03:53.630 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:03:53.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:03:53.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:03:53.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:03:53.952 INFO:tasks.workunit.client.0.smithi129.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2029505744,v1:127.0.0.1:6803/2029505744] [v2:127.0.0.1:6804/2029505744,v1:127.0.0.1:6805/2029505744] exists,up fbcb1bb8-89a6-4108-b8af-81c759b5546a 2024-03-25T23:03:53.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:03:53.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:03:53.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:03:53.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-25T23:03:53.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=false 2024-03-25T23:03:53.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-25T23:03:53.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:03:53.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-25T23:03:53.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:03:53.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/1 2024-03-25T23:03:53.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4de2c693-e629-4160-940c-ecd79a0b43d9 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-25T23:03:53.957 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:03:53.957 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:03:53.957 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:03:53.957 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2024-03-25T23:03:53.957 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2024-03-25T23:03:53.957 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:03:53.957 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:03:53.957 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-25T23:03:53.957 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:03:53.957 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:03:53.957 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:03:53.957 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:03:53.957 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:03:53.957 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36561 2024-03-25T23:03:53.958 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36561/$cluster-$name.asok' 2024-03-25T23:03:53.958 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36561/$cluster-$name.asok' 2024-03-25T23:03:53.958 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:03:53.958 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:03:53.958 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:03:53.958 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-25T23:03:53.958 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-25T23:03:53.958 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:03:53.958 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:03:53.958 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:03:53.958 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:03:53.958 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:03:53.958 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-03-25T23:03:53.958 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/1 2024-03-25T23:03:53.959 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:03:53.959 INFO:tasks.workunit.client.0.smithi129.stdout:add osd1 d602ff66-83d9-4a03-a36f-1a3af9b79744 2024-03-25T23:03:53.959 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d602ff66-83d9-4a03-a36f-1a3af9b79744 2024-03-25T23:03:53.960 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 d602ff66-83d9-4a03-a36f-1a3af9b79744' 2024-03-25T23:03:53.960 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:03:53.976 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDZAgJm38AqOhAApqp9ixTnRp/uH08zsixHdA== 2024-03-25T23:03:53.976 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDZAgJm38AqOhAApqp9ixTnRp/uH08zsixHdA=="}' 2024-03-25T23:03:53.976 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d602ff66-83d9-4a03-a36f-1a3af9b79744 -i td/osd-recovery-scrub/1/new.json 2024-03-25T23:03:54.382 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:03:54.397 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/1/new.json 2024-03-25T23:03:54.398 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=4de2c693-e629-4160-940c-ecd79a0b43d9 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36561/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQDZAgJm38AqOhAApqp9ixTnRp/uH08zsixHdA== --osd-uuid d602ff66-83d9-4a03-a36f-1a3af9b79744 2024-03-25T23:03:54.435 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:54.432+0000 7f5a10be97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:03:54.488 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:54.484+0000 7f5a10be97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:03:54.489 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:54.488+0000 7f5a10be97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:03:54.490 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:54.488+0000 7f5a10be97c0 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-03-25T23:03:54.490 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:54.488+0000 7f5a10be97c0 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-03-25T23:03:54.490 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:54.488+0000 7f5a10be97c0 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-03-25T23:03:54.563 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:03:54.560+0000 7f5a10be97c0 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2024-03-25T23:04:01.642 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2024-03-25T23:04:01.642 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:04:01.644 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd1 key to auth repository 2024-03-25T23:04:01.644 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-25T23:04:01.644 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:04:02.029 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.1 2024-03-25T23:04:02.030 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-25T23:04:02.030 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=4de2c693-e629-4160-940c-ecd79a0b43d9 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36561/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-03-25T23:04:02.031 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:04:02.031 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:04:02.031 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:04:02.061 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:04:02.056+0000 7f28a5f9e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:04:02.062 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:04:02.060+0000 7f28a5f9e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:04:02.064 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:04:02.060+0000 7f28a5f9e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:04:02.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-25T23:04:02.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:04:02.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T23:04:02.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:04:02.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:04:02.348 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:04:02.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:04:02.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:04:02.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:04:02.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:04:02.611 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:04:02.608+0000 7f28a5f9e7c0 -1 Falling back to public interface 2024-03-25T23:04:02.708 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:04:03.709 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:04:03.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:04:03.710 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:04:03.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:04:03.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:04:03.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:04:04.011 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:04:05.012 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:04:05.013 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:04:05.013 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:04:05.014 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:04:05.014 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:04:05.014 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:04:05.333 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:04:05.407 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:04:05.404+0000 7f28a5f9e7c0 -1 osd.1 0 log_to_monitors true 2024-03-25T23:04:06.334 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:04:06.335 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:04:06.335 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:04:06.336 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:04:06.336 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:04:06.336 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:04:06.649 INFO:tasks.workunit.client.0.smithi129.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2196124109,v1:127.0.0.1:6811/2196124109] [v2:127.0.0.1:6812/2196124109,v1:127.0.0.1:6813/2196124109] exists,up d602ff66-83d9-4a03-a36f-1a3af9b79744 2024-03-25T23:04:06.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:04:06.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:04:06.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:04:06.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-25T23:04:06.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=false 2024-03-25T23:04:06.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-25T23:04:06.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:04:06.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-25T23:04:06.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:04:06.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/2 2024-03-25T23:04:06.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4de2c693-e629-4160-940c-ecd79a0b43d9 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-25T23:04:06.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:04:06.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:04:06.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:04:06.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2024-03-25T23:04:06.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2024-03-25T23:04:06.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:04:06.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:04:06.655 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-25T23:04:06.655 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:04:06.655 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:04:06.655 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:04:06.655 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:04:06.655 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:04:06.655 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36561 2024-03-25T23:04:06.655 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36561/$cluster-$name.asok' 2024-03-25T23:04:06.655 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36561/$cluster-$name.asok' 2024-03-25T23:04:06.655 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:04:06.656 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:04:06.656 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:04:06.656 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-25T23:04:06.656 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-25T23:04:06.656 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:04:06.656 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:04:06.656 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:04:06.656 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:04:06.656 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:04:06.656 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-03-25T23:04:06.656 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/2 2024-03-25T23:04:06.657 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:04:06.657 INFO:tasks.workunit.client.0.smithi129.stdout:add osd2 1570a643-b838-44fa-bfdb-865504bdff27 2024-03-25T23:04:06.657 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1570a643-b838-44fa-bfdb-865504bdff27 2024-03-25T23:04:06.657 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 1570a643-b838-44fa-bfdb-865504bdff27' 2024-03-25T23:04:06.657 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:04:06.677 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDmAgJmZIVVKBAA8NvUk3bGdPiS49/32S1tHg== 2024-03-25T23:04:06.677 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDmAgJmZIVVKBAA8NvUk3bGdPiS49/32S1tHg=="}' 2024-03-25T23:04:06.677 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1570a643-b838-44fa-bfdb-865504bdff27 -i td/osd-recovery-scrub/2/new.json 2024-03-25T23:04:07.092 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:04:07.109 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/2/new.json 2024-03-25T23:04:07.110 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=4de2c693-e629-4160-940c-ecd79a0b43d9 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36561/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQDmAgJmZIVVKBAA8NvUk3bGdPiS49/32S1tHg== --osd-uuid 1570a643-b838-44fa-bfdb-865504bdff27 2024-03-25T23:04:07.142 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:04:07.140+0000 7f39ad5a37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:04:07.193 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:04:07.192+0000 7f39ad5a37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:04:07.194 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:04:07.192+0000 7f39ad5a37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:04:07.195 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:04:07.192+0000 7f39ad5a37c0 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-03-25T23:04:07.195 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:04:07.192+0000 7f39ad5a37c0 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-03-25T23:04:07.195 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:04:07.192+0000 7f39ad5a37c0 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-03-25T23:04:07.279 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:04:07.276+0000 7f39ad5a37c0 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2024-03-25T23:04:14.300 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2024-03-25T23:04:14.300 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:04:14.301 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd2 key to auth repository 2024-03-25T23:04:14.301 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-25T23:04:14.301 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:04:14.687 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-25T23:04:14.687 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.2 2024-03-25T23:04:14.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=4de2c693-e629-4160-940c-ecd79a0b43d9 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36561/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-03-25T23:04:14.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:04:14.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:04:14.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:04:14.720 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:04:14.715+0000 7fa54a8807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:04:14.722 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:04:14.719+0000 7fa54a8807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:04:14.723 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:04:14.719+0000 7fa54a8807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:04:15.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-25T23:04:15.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:04:15.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-25T23:04:15.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:04:15.018 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:04:15.018 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:04:15.018 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:04:15.018 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:04:15.018 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:04:15.018 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:04:15.275 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:04:15.271+0000 7fa54a8807c0 -1 Falling back to public interface 2024-03-25T23:04:15.357 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:04:16.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:04:16.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:04:16.359 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:04:16.360 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:04:16.360 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:04:16.360 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:04:16.701 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:04:17.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:04:17.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:04:17.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:04:17.703 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:04:17.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:04:17.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:04:17.897 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:04:17.895+0000 7fa54a8807c0 -1 osd.2 0 log_to_monitors true 2024-03-25T23:04:18.024 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:04:19.025 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:04:19.026 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:04:19.026 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:04:19.026 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:04:19.026 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:04:19.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:04:19.377 INFO:tasks.workunit.client.0.smithi129.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1348070892,v1:127.0.0.1:6819/1348070892] [v2:127.0.0.1:6820/1348070892,v1:127.0.0.1:6821/1348070892] exists,up 1570a643-b838-44fa-bfdb-865504bdff27 2024-03-25T23:04:19.381 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:04:19.381 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:04:19.381 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:04:19.381 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-25T23:04:19.381 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=false 2024-03-25T23:04:19.381 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-25T23:04:19.381 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:04:19.381 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-03-25T23:04:19.381 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:04:19.381 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/3 2024-03-25T23:04:19.381 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4de2c693-e629-4160-940c-ecd79a0b43d9 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-25T23:04:19.381 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:04:19.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:04:19.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:04:19.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2024-03-25T23:04:19.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2024-03-25T23:04:19.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:04:19.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:04:19.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-25T23:04:19.382 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:04:19.382 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:04:19.382 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:04:19.382 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:04:19.382 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:04:19.382 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36561 2024-03-25T23:04:19.383 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36561/$cluster-$name.asok' 2024-03-25T23:04:19.383 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36561/$cluster-$name.asok' 2024-03-25T23:04:19.383 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:04:19.383 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:04:19.383 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:04:19.383 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-25T23:04:19.383 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-25T23:04:19.383 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:04:19.383 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:04:19.383 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:04:19.383 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:04:19.383 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:04:19.383 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-03-25T23:04:19.383 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/3 2024-03-25T23:04:19.384 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:04:19.384 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ea51d0d7-b1a5-474e-a1b2-1ec9d06ffbd9 2024-03-25T23:04:19.384 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 ea51d0d7-b1a5-474e-a1b2-1ec9d06ffbd9' 2024-03-25T23:04:19.385 INFO:tasks.workunit.client.0.smithi129.stdout:add osd3 ea51d0d7-b1a5-474e-a1b2-1ec9d06ffbd9 2024-03-25T23:04:19.385 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:04:19.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDzAgJmoFA4GBAARmDpBs8D60DVt13fF9HWJA== 2024-03-25T23:04:19.407 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDzAgJmoFA4GBAARmDpBs8D60DVt13fF9HWJA=="}' 2024-03-25T23:04:19.407 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ea51d0d7-b1a5-474e-a1b2-1ec9d06ffbd9 -i td/osd-recovery-scrub/3/new.json 2024-03-25T23:04:19.783 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:04:19.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/3/new.json 2024-03-25T23:04:19.804 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=4de2c693-e629-4160-940c-ecd79a0b43d9 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36561/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQDzAgJmoFA4GBAARmDpBs8D60DVt13fF9HWJA== --osd-uuid ea51d0d7-b1a5-474e-a1b2-1ec9d06ffbd9 2024-03-25T23:04:19.839 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:04:19.835+0000 7fc4195a37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:04:19.893 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:04:19.891+0000 7fc4195a37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:04:19.895 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:04:19.891+0000 7fc4195a37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:04:19.895 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:04:19.891+0000 7fc4195a37c0 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-03-25T23:04:19.895 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:04:19.891+0000 7fc4195a37c0 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-03-25T23:04:19.896 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:04:19.891+0000 7fc4195a37c0 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-03-25T23:04:19.974 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:04:19.971+0000 7fc4195a37c0 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2024-03-25T23:04:26.986 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2024-03-25T23:04:26.986 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:04:26.987 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd3 key to auth repository 2024-03-25T23:04:26.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-03-25T23:04:26.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:04:27.414 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.3 2024-03-25T23:04:27.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-03-25T23:04:27.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=4de2c693-e629-4160-940c-ecd79a0b43d9 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36561/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-03-25T23:04:27.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:04:27.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:04:27.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:04:27.440 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:04:27.435+0000 7fd4d6ab47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:04:27.442 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:04:27.439+0000 7fd4d6ab47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:04:27.443 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:04:27.439+0000 7fd4d6ab47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:04:27.745 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-03-25T23:04:27.745 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:04:27.745 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-25T23:04:27.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:04:27.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:04:27.746 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:04:27.747 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:04:27.747 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:04:27.747 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:04:27.747 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:04:28.003 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:04:27.999+0000 7fd4d6ab47c0 -1 Falling back to public interface 2024-03-25T23:04:28.075 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:04:29.076 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:04:29.076 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:04:29.076 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:04:29.077 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:04:29.077 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:04:29.077 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:04:29.396 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:04:30.398 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:04:30.398 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:04:30.399 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:04:30.399 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:04:30.399 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:04:30.399 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:04:30.733 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:04:30.763 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:04:30.759+0000 7fd4d6ab47c0 -1 osd.3 0 log_to_monitors true 2024-03-25T23:04:31.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:04:31.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:04:31.735 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:04:31.736 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:04:31.736 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:04:31.736 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:04:32.058 INFO:tasks.workunit.client.0.smithi129.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2963333175,v1:127.0.0.1:6827/2963333175] [v2:127.0.0.1:6828/2963333175,v1:127.0.0.1:6829/2963333175] exists,up ea51d0d7-b1a5-474e-a1b2-1ec9d06ffbd9 2024-03-25T23:04:32.059 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:04:32.059 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:04:32.059 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:04:32.059 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:61: TEST_recovery_scrub_1: create_pool test 1 1 2024-03-25T23:04:32.059 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-03-25T23:04:32.474 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'test' created 2024-03-25T23:04:32.492 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T23:04:33.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:62: TEST_recovery_scrub_1: wait_for_clean 2024-03-25T23:04:33.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:04:33.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:04:33.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:04:33.495 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:04:33.495 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:04:33.496 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:04:33.496 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:04:33.496 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:04:33.496 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:04:33.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:04:33.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:04:33.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:04:33.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:04:33.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:04:33.616 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:04:34.041 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:04:34.041 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:04:34.041 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:04:34.041 INFO:tasks.workunit.client.0.smithi129.stderr:3' 2024-03-25T23:04:34.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:04:34.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:04:34.042 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:04:34.190 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836490 2024-03-25T23:04:34.190 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836490 2024-03-25T23:04:34.190 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836490' 2024-03-25T23:04:34.190 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:04:34.190 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:04:34.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672968 2024-03-25T23:04:34.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672968 2024-03-25T23:04:34.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836490 1-42949672968' 2024-03-25T23:04:34.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:04:34.332 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:04:34.469 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509445 2024-03-25T23:04:34.470 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509445 2024-03-25T23:04:34.470 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836490 1-42949672968 2-64424509445' 2024-03-25T23:04:34.470 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:04:34.470 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:04:34.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345922 2024-03-25T23:04:34.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345922 2024-03-25T23:04:34.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836490 1-42949672968 2-64424509445 3-85899345922' 2024-03-25T23:04:34.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:04:34.652 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836490 2024-03-25T23:04:34.652 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:04:34.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:04:34.653 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836490 2024-03-25T23:04:34.654 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:04:34.655 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836490 2024-03-25T23:04:34.655 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2024-03-25T23:04:34.656 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 21474836490 2024-03-25T23:04:34.656 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:04:34.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836490 -lt 21474836490 2024-03-25T23:04:34.968 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:04:34.968 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672968 2024-03-25T23:04:34.969 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:04:34.970 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:04:34.971 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672968 2024-03-25T23:04:34.971 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:04:34.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672968 2024-03-25T23:04:34.973 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 42949672968 2024-03-25T23:04:34.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2024-03-25T23:04:34.973 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:04:35.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672968 -lt 42949672968 2024-03-25T23:04:35.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:04:35.304 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:04:35.304 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509445 2024-03-25T23:04:35.305 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:04:35.305 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509445 2024-03-25T23:04:35.306 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:04:35.306 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509445 2024-03-25T23:04:35.307 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 64424509445 2024-03-25T23:04:35.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-03-25T23:04:35.307 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:04:35.644 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509444 -lt 64424509445 2024-03-25T23:04:35.644 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:04:36.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:04:36.647 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:04:36.955 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509445 -lt 64424509445 2024-03-25T23:04:36.955 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:04:36.955 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345922 2024-03-25T23:04:36.956 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:04:36.957 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:04:36.958 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345922 2024-03-25T23:04:36.958 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:04:36.959 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345922 2024-03-25T23:04:36.959 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345922' 2024-03-25T23:04:36.960 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.3 seq 85899345922 2024-03-25T23:04:36.960 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:04:37.293 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345922 -lt 85899345922 2024-03-25T23:04:37.294 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:04:37.294 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:04:37.294 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:04:37.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-25T23:04:37.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:04:37.677 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:04:37.677 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:04:37.677 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:04:37.677 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:04:37.677 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:04:37.677 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:04:38.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-25T23:04:38.017 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:04:38.018 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:04:38.018 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:04:38.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-25T23:04:38.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:04:38.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:04:38.395 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: grep '^pool.*['\'']test['\'']' 2024-03-25T23:04:38.395 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: ceph osd dump 2024-03-25T23:04:38.395 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: awk '{ print $2 }' 2024-03-25T23:04:38.747 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: poolid=1 2024-03-25T23:04:38.747 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:65: TEST_recovery_scrub_1: ceph pg dump pgs 2024-03-25T23:04:39.061 INFO:tasks.workunit.client.0.smithi129.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-25T23:04:39.061 INFO:tasks.workunit.client.0.smithi129.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T23:04:33.705774+0000 0'0 24:12 [1] 1 [1] 1 0'0 2024-03-25T23:04:32.407426+0000 0'0 2024-03-25T23:04:32.407426+0000 0 0 periodic scrub scheduled @ 2024-03-27T06:38:11.115325+0000 0 0 2024-03-25T23:04:39.061 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:04:39.061 INFO:tasks.workunit.client.0.smithi129.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-25T23:04:39.061 INFO:tasks.workunit.client.0.smithi129.stderr:dumped pgs 2024-03-25T23:04:39.075 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub_1: dd if=/dev/urandom of=testdata.36561 bs=1M count=50 2024-03-25T23:04:39.245 INFO:tasks.workunit.client.0.smithi129.stderr:50+0 records in 2024-03-25T23:04:39.245 INFO:tasks.workunit.client.0.smithi129.stderr:50+0 records out 2024-03-25T23:04:39.245 INFO:tasks.workunit.client.0.smithi129.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.16831 s, 312 MB/s 2024-03-25T23:04:39.246 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: seq 1 100 2024-03-25T23:04:39.247 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:04:39.247 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj1 testdata.36561 2024-03-25T23:04:41.204 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:04:41.204 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj2 testdata.36561 2024-03-25T23:04:42.936 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:04:42.936 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj3 testdata.36561 2024-03-25T23:04:44.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:04:44.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj4 testdata.36561 2024-03-25T23:04:46.475 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:04:46.475 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj5 testdata.36561 2024-03-25T23:04:48.273 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:04:48.273 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj6 testdata.36561 2024-03-25T23:04:50.111 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:04:50.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj7 testdata.36561 2024-03-25T23:04:51.944 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:04:51.944 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj8 testdata.36561 2024-03-25T23:04:53.683 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:04:53.683 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj9 testdata.36561 2024-03-25T23:04:55.465 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:04:55.466 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj10 testdata.36561 2024-03-25T23:04:57.306 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:04:57.306 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj11 testdata.36561 2024-03-25T23:04:59.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:04:59.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj12 testdata.36561 2024-03-25T23:05:01.094 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:05:01.094 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj13 testdata.36561 2024-03-25T23:05:02.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:05:02.818 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj14 testdata.36561 2024-03-25T23:05:04.673 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:05:04.673 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj15 testdata.36561 2024-03-25T23:05:06.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:05:06.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj16 testdata.36561 2024-03-25T23:05:08.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:05:08.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj17 testdata.36561 2024-03-25T23:05:10.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:05:10.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj18 testdata.36561 2024-03-25T23:05:12.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:05:12.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj19 testdata.36561 2024-03-25T23:05:14.274 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:05:14.274 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj20 testdata.36561 2024-03-25T23:05:16.146 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:05:16.147 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj21 testdata.36561 2024-03-25T23:05:17.954 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:05:17.954 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj22 testdata.36561 2024-03-25T23:05:19.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:05:19.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj23 testdata.36561 2024-03-25T23:05:21.551 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:05:21.551 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj24 testdata.36561 2024-03-25T23:05:23.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:05:23.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj25 testdata.36561 2024-03-25T23:05:24.947 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:05:24.947 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj26 testdata.36561 2024-03-25T23:05:26.637 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:05:26.637 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj27 testdata.36561 2024-03-25T23:05:28.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:05:28.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj28 testdata.36561 2024-03-25T23:05:30.092 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:05:30.092 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj29 testdata.36561 2024-03-25T23:05:32.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:05:32.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj30 testdata.36561 2024-03-25T23:05:33.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:05:33.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj31 testdata.36561 2024-03-25T23:05:35.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:05:35.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj32 testdata.36561 2024-03-25T23:05:37.411 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:05:37.411 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj33 testdata.36561 2024-03-25T23:05:39.141 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:05:39.141 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj34 testdata.36561 2024-03-25T23:05:40.877 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:05:40.877 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj35 testdata.36561 2024-03-25T23:05:42.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:05:42.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj36 testdata.36561 2024-03-25T23:05:44.220 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:05:44.220 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj37 testdata.36561 2024-03-25T23:05:45.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:05:45.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj38 testdata.36561 2024-03-25T23:05:47.641 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:05:47.642 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj39 testdata.36561 2024-03-25T23:05:49.423 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:05:49.423 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj40 testdata.36561 2024-03-25T23:05:51.020 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:05:51.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj41 testdata.36561 2024-03-25T23:05:52.694 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:05:52.694 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj42 testdata.36561 2024-03-25T23:05:54.259 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:05:54.259 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj43 testdata.36561 2024-03-25T23:05:55.890 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:05:55.890 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj44 testdata.36561 2024-03-25T23:05:57.522 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:05:57.522 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj45 testdata.36561 2024-03-25T23:05:59.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:05:59.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj46 testdata.36561 2024-03-25T23:06:00.675 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:06:00.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj47 testdata.36561 2024-03-25T23:06:02.583 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:06:02.583 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj48 testdata.36561 2024-03-25T23:06:04.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:06:04.198 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj49 testdata.36561 2024-03-25T23:06:05.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:06:05.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj50 testdata.36561 2024-03-25T23:06:07.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:06:07.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj51 testdata.36561 2024-03-25T23:06:09.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:06:09.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj52 testdata.36561 2024-03-25T23:06:10.765 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:06:10.765 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj53 testdata.36561 2024-03-25T23:06:12.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:06:12.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj54 testdata.36561 2024-03-25T23:06:14.012 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:06:14.012 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj55 testdata.36561 2024-03-25T23:06:15.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:06:15.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj56 testdata.36561 2024-03-25T23:06:17.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:06:17.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj57 testdata.36561 2024-03-25T23:06:18.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:06:18.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj58 testdata.36561 2024-03-25T23:06:20.479 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:06:20.479 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj59 testdata.36561 2024-03-25T23:06:22.117 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:06:22.117 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj60 testdata.36561 2024-03-25T23:06:23.809 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:06:23.809 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj61 testdata.36561 2024-03-25T23:06:25.440 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:06:25.441 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj62 testdata.36561 2024-03-25T23:06:27.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:06:27.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj63 testdata.36561 2024-03-25T23:06:28.669 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:06:28.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj64 testdata.36561 2024-03-25T23:06:30.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:06:30.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj65 testdata.36561 2024-03-25T23:06:31.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:06:31.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj66 testdata.36561 2024-03-25T23:06:33.866 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:06:33.866 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj67 testdata.36561 2024-03-25T23:06:35.606 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:06:35.606 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj68 testdata.36561 2024-03-25T23:06:37.362 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:06:37.362 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj69 testdata.36561 2024-03-25T23:06:39.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:06:39.035 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj70 testdata.36561 2024-03-25T23:06:40.889 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:06:40.889 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj71 testdata.36561 2024-03-25T23:06:42.679 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:06:42.679 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj72 testdata.36561 2024-03-25T23:06:44.393 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:06:44.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj73 testdata.36561 2024-03-25T23:06:46.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:06:46.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj74 testdata.36561 2024-03-25T23:06:47.965 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:06:47.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj75 testdata.36561 2024-03-25T23:06:49.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:06:49.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj76 testdata.36561 2024-03-25T23:06:51.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:06:51.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj77 testdata.36561 2024-03-25T23:06:53.351 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:06:53.351 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj78 testdata.36561 2024-03-25T23:06:55.072 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:06:55.072 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj79 testdata.36561 2024-03-25T23:06:56.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:06:56.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj80 testdata.36561 2024-03-25T23:06:58.502 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:06:58.502 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj81 testdata.36561 2024-03-25T23:07:00.258 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:07:00.258 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj82 testdata.36561 2024-03-25T23:07:02.073 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:07:02.073 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj83 testdata.36561 2024-03-25T23:07:04.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:07:04.022 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj84 testdata.36561 2024-03-25T23:07:05.886 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:07:05.886 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj85 testdata.36561 2024-03-25T23:07:07.641 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:07:07.641 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj86 testdata.36561 2024-03-25T23:07:09.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:07:09.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj87 testdata.36561 2024-03-25T23:07:11.053 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:07:11.053 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj88 testdata.36561 2024-03-25T23:07:12.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:07:12.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj89 testdata.36561 2024-03-25T23:07:14.456 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:07:14.456 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj90 testdata.36561 2024-03-25T23:07:16.220 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:07:16.220 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj91 testdata.36561 2024-03-25T23:07:17.977 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:07:17.977 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj92 testdata.36561 2024-03-25T23:07:19.859 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:07:19.859 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj93 testdata.36561 2024-03-25T23:07:21.624 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:07:21.624 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj94 testdata.36561 2024-03-25T23:07:23.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:07:23.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj95 testdata.36561 2024-03-25T23:07:25.218 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:07:25.218 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj96 testdata.36561 2024-03-25T23:07:26.941 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:07:26.941 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj97 testdata.36561 2024-03-25T23:07:28.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:07:28.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj98 testdata.36561 2024-03-25T23:07:30.487 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:07:30.488 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj99 testdata.36561 2024-03-25T23:07:32.335 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-25T23:07:32.335 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj100 testdata.36561 2024-03-25T23:07:34.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:72: TEST_recovery_scrub_1: rm -f testdata.36561 2024-03-25T23:07:34.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:74: TEST_recovery_scrub_1: ceph osd pool set test size 4 2024-03-25T23:07:34.718 INFO:tasks.workunit.client.0.smithi129.stderr:set pool 1 size to 4 2024-03-25T23:07:34.738 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:77: TEST_recovery_scrub_1: set -o pipefail 2024-03-25T23:07:34.739 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: count=0 2024-03-25T23:07:34.739 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-03-25T23:07:34.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-03-25T23:07:34.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-03-25T23:07:34.741 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-03-25T23:07:35.016 INFO:tasks.workunit.client.0.smithi129.stderr:dumped pgs 2024-03-25T23:07:35.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-03-25T23:07:37.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 0 -eq 10 2024-03-25T23:07:37.035 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 0 + 1 2024-03-25T23:07:37.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=1 2024-03-25T23:07:37.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-03-25T23:07:37.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-03-25T23:07:37.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-03-25T23:07:37.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-03-25T23:07:37.366 INFO:tasks.workunit.client.0.smithi129.stderr:dumped pgs 2024-03-25T23:07:37.384 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-03-25T23:07:39.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 1 -eq 10 2024-03-25T23:07:39.386 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 1 + 1 2024-03-25T23:07:39.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=2 2024-03-25T23:07:39.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-03-25T23:07:39.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-03-25T23:07:39.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-03-25T23:07:39.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-03-25T23:07:39.916 INFO:tasks.workunit.client.0.smithi129.stderr:dumped pgs 2024-03-25T23:07:39.933 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-03-25T23:07:41.934 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 2 -eq 10 2024-03-25T23:07:41.935 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 2 + 1 2024-03-25T23:07:41.936 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=3 2024-03-25T23:07:41.937 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-03-25T23:07:41.938 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-03-25T23:07:41.939 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-03-25T23:07:41.939 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-03-25T23:07:42.247 INFO:tasks.workunit.client.0.smithi129.stderr:dumped pgs 2024-03-25T23:07:42.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-03-25T23:07:44.264 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 3 -eq 10 2024-03-25T23:07:44.266 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 3 + 1 2024-03-25T23:07:44.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=4 2024-03-25T23:07:44.268 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-03-25T23:07:44.269 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-03-25T23:07:44.269 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-03-25T23:07:44.269 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-03-25T23:07:44.562 INFO:tasks.workunit.client.0.smithi129.stderr:dumped pgs 2024-03-25T23:07:44.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:84: TEST_recovery_scrub_1: break 2024-03-25T23:07:44.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:94: TEST_recovery_scrub_1: set +o pipefail 2024-03-25T23:07:44.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:95: TEST_recovery_scrub_1: ceph pg dump pgs 2024-03-25T23:07:44.857 INFO:tasks.workunit.client.0.smithi129.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-25T23:07:44.857 INFO:tasks.workunit.client.0.smithi129.stdout:1.0 100 0 297 0 0 5242880000 0 0 1300 0 1300 active+recovering+undersized+degraded+remapped 2024-03-25T23:07:38.287088+0000 24'1300 27:1328 [1,0,2,3] 1 [1,0] 1 0'0 2024-03-25T23:04:32.407426+0000 0'0 2024-03-25T23:04:32.407426+0000 0 0 no scrub is scheduled 0 0 2024-03-25T23:07:44.857 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:07:44.857 INFO:tasks.workunit.client.0.smithi129.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-25T23:07:44.858 INFO:tasks.workunit.client.0.smithi129.stderr:dumped pgs 2024-03-25T23:07:44.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:97: TEST_recovery_scrub_1: sleep 10 2024-03-25T23:07:54.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:99: TEST_recovery_scrub_1: kill_daemons td/osd-recovery-scrub 2024-03-25T23:07:54.876 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:07:54.876 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:07:54.876 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:07:54.876 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:07:54.876 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:08:00.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:08:00.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:101: TEST_recovery_scrub_1: declare -a err_strings 2024-03-25T23:08:00.198 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub_1: err_strings[0]='recovery in progress. Only high priority scrubs allowed.' 2024-03-25T23:08:00.198 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: expr 4 - 1 2024-03-25T23:08:00.199 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: seq 0 3 2024-03-25T23:08:00.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-25T23:08:00.200 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.0.log 2024-03-25T23:08:00.203 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:07:39.012+0000 7f76a3ab3640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-25T23:08:00.203 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:07:45.940+0000 7f76a3ab3640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-25T23:08:00.203 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:07:47.912+0000 7f76a3ab3640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-25T23:08:00.203 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:07:48.884+0000 7f76a3ab3640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-25T23:08:00.203 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:07:49.920+0000 7f76a3ab3640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-25T23:08:00.203 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:07:52.928+0000 7f76a3ab3640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-25T23:08:00.203 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:07:53.880+0000 7f76a3ab3640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-25T23:08:00.203 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:07:55.864+0000 7f76a3ab3640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-25T23:08:00.203 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:07:56.856+0000 7f76a3ab3640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-25T23:08:00.204 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-25T23:08:00.204 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.1.log 2024-03-25T23:08:00.217 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:07:40.768+0000 7f289ef52640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-25T23:08:00.218 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:07:41.740+0000 7f289ef52640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-25T23:08:00.218 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:07:42.732+0000 7f289ef52640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-25T23:08:00.218 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:07:43.708+0000 7f289ef52640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-25T23:08:00.218 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:07:52.664+0000 7f289ef52640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-25T23:08:00.218 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-25T23:08:00.218 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.2.log 2024-03-25T23:08:00.221 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:07:45.820+0000 7fa543834640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-25T23:08:00.221 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:07:46.776+0000 7fa543834640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-25T23:08:00.221 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:07:49.816+0000 7fa543834640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-25T23:08:00.221 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:07:52.776+0000 7fa543834640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-25T23:08:00.221 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:07:53.820+0000 7fa543834640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-25T23:08:00.221 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:07:54.844+0000 7fa543834640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-25T23:08:00.221 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:07:58.832+0000 7fa543834640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-25T23:08:00.222 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-25T23:08:00.222 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.3.log 2024-03-25T23:08:00.224 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:07:38.612+0000 7fd4cfa68640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-25T23:08:00.224 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:07:39.644+0000 7fd4cfa68640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-25T23:08:00.224 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:07:40.664+0000 7fd4cfa68640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-25T23:08:00.224 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:07:42.664+0000 7fd4cfa68640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-25T23:08:00.224 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:07:44.664+0000 7fd4cfa68640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-25T23:08:00.224 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:07:46.648+0000 7fd4cfa68640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-25T23:08:00.224 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:07:50.552+0000 7fd4cfa68640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-25T23:08:00.224 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:07:53.560+0000 7fd4cfa68640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-25T23:08:00.224 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:07:54.572+0000 7fd4cfa68640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-25T23:08:00.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: for err_string in "${err_strings[@]}" 2024-03-25T23:08:00.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:110: TEST_recovery_scrub_1: found=false 2024-03-25T23:08:00.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:111: TEST_recovery_scrub_1: count=0 2024-03-25T23:08:00.225 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: expr 4 - 1 2024-03-25T23:08:00.226 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: seq 0 3 2024-03-25T23:08:00.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-25T23:08:00.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.0.log 2024-03-25T23:08:00.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-03-25T23:08:00.230 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 0 + 1 2024-03-25T23:08:00.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=1 2024-03-25T23:08:00.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-25T23:08:00.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.1.log 2024-03-25T23:08:00.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-03-25T23:08:00.246 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 1 + 1 2024-03-25T23:08:00.248 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=2 2024-03-25T23:08:00.248 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-25T23:08:00.248 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.2.log 2024-03-25T23:08:00.250 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-03-25T23:08:00.250 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 2 + 1 2024-03-25T23:08:00.252 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=3 2024-03-25T23:08:00.252 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-25T23:08:00.252 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.3.log 2024-03-25T23:08:00.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-03-25T23:08:00.255 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 3 + 1 2024-03-25T23:08:00.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=4 2024-03-25T23:08:00.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:120: TEST_recovery_scrub_1: '[' true = false ']' 2024-03-25T23:08:00.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:124: TEST_recovery_scrub_1: '[' 4 -eq 4 ']' 2024-03-25T23:08:00.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:127: TEST_recovery_scrub_1: teardown td/osd-recovery-scrub 2024-03-25T23:08:00.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-03-25T23:08:00.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T23:08:00.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-03-25T23:08:00.257 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:08:00.257 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:08:00.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:08:00.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:08:00.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:08:00.264 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:08:00.264 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T23:08:00.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T23:08:00.265 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T23:08:00.266 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T23:08:00.266 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T23:08:00.266 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T23:08:00.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:08:00.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T23:08:00.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T23:08:00.267 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:08:00.268 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T23:08:00.269 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T23:08:00.269 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-03-25T23:08:00.293 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T23:08:00.293 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:08:00.293 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36561 2024-03-25T23:08:00.293 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36561 2024-03-25T23:08:00.294 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T23:08:00.294 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T23:08:00.294 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:129: TEST_recovery_scrub_1: '[' 0 '!=' 0 ']' 2024-03-25T23:08:00.294 INFO:tasks.workunit.client.0.smithi129.stdout:TEST PASSED 2024-03-25T23:08:00.296 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:135: TEST_recovery_scrub_1: echo 'TEST PASSED' 2024-03-25T23:08:00.296 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:136: TEST_recovery_scrub_1: return 0 2024-03-25T23:08:00.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: for func in $funcs 2024-03-25T23:08:00.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: TEST_recovery_scrub_2 td/osd-recovery-scrub 2024-03-25T23:08:00.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:234: TEST_recovery_scrub_2: local dir=td/osd-recovery-scrub 2024-03-25T23:08:00.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:235: TEST_recovery_scrub_2: local poolname=test 2024-03-25T23:08:00.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:237: TEST_recovery_scrub_2: TESTDATA=testdata.36561 2024-03-25T23:08:00.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:238: TEST_recovery_scrub_2: OSDS=8 2024-03-25T23:08:00.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:239: TEST_recovery_scrub_2: PGS=32 2024-03-25T23:08:00.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:240: TEST_recovery_scrub_2: OBJECTS=40 2024-03-25T23:08:00.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: setup td/osd-recovery-scrub 2024-03-25T23:08:00.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-scrub 2024-03-25T23:08:00.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-scrub 2024-03-25T23:08:00.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-03-25T23:08:00.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T23:08:00.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-03-25T23:08:00.298 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:08:00.298 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:08:00.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:08:00.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:08:00.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:08:00.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:08:00.298 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T23:08:00.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T23:08:00.299 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T23:08:00.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T23:08:00.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T23:08:00.299 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T23:08:00.300 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:08:00.300 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T23:08:00.300 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T23:08:00.300 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:08:00.300 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T23:08:00.301 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T23:08:00.301 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-03-25T23:08:00.302 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T23:08:00.302 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:08:00.303 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36561 2024-03-25T23:08:00.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36561 2024-03-25T23:08:00.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T23:08:00.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T23:08:00.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-scrub 2024-03-25T23:08:00.304 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-25T23:08:00.304 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:08:00.305 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36561 2024-03-25T23:08:00.305 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.36561 2024-03-25T23:08:00.306 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-25T23:08:00.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-25T23:08:00.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-25T23:08:00.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2024-03-25T23:08:00.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:243: TEST_recovery_scrub_2: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-03-25T23:08:00.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-scrub 2024-03-25T23:08:00.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-25T23:08:00.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-25T23:08:00.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-25T23:08:00.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-scrub/a 2024-03-25T23:08:00.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-03-25T23:08:00.831 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-25T23:08:00.831 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:08:00.832 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:08:00.832 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:08:00.832 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:08:00.832 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36561 2024-03-25T23:08:00.833 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36561/$cluster-$name.asok' 2024-03-25T23:08:00.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.36561/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-03-25T23:08:01.076 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-25T23:08:01.077 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-25T23:08:01.077 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T23:08:01.077 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T23:08:01.078 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-25T23:08:01.078 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-25T23:08:01.078 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T23:08:01.078 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T23:08:01.078 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T23:08:01.079 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T23:08:01.079 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:08:01.079 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36561 2024-03-25T23:08:01.079 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36561/ceph-mon.a.asok 2024-03-25T23:08:01.079 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T23:08:01.080 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.36561/ceph-mon.a.asok config get fsid 2024-03-25T23:08:01.196 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-25T23:08:01.196 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T23:08:01.197 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T23:08:01.197 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-25T23:08:01.197 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-25T23:08:01.198 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T23:08:01.198 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T23:08:01.198 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T23:08:01.198 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T23:08:01.198 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:08:01.198 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36561 2024-03-25T23:08:01.198 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36561/ceph-mon.a.asok 2024-03-25T23:08:01.199 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T23:08:01.199 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.36561/ceph-mon.a.asok config get mon_host 2024-03-25T23:08:01.308 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_mgr td/osd-recovery-scrub x 2024-03-25T23:08:01.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-scrub 2024-03-25T23:08:01.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-25T23:08:01.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-25T23:08:01.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-25T23:08:01.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-scrub/x 2024-03-25T23:08:01.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-25T23:08:02.194 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-25T23:08:02.194 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:08:02.194 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:08:02.195 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:08:02.195 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:08:02.195 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36561 2024-03-25T23:08:02.195 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36561/$cluster-$name.asok' 2024-03-25T23:08:02.196 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T23:08:02.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.36561/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T23:08:02.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:245: TEST_recovery_scrub_2: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 ' 2024-03-25T23:08:02.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:246: TEST_recovery_scrub_2: ceph_osd_args+='--osd_scrub_backoff_ratio=0 ' 2024-03-25T23:08:02.233 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:247: TEST_recovery_scrub_2: ceph_osd_args+='--osd_stats_update_period_not_scrubbing=3 ' 2024-03-25T23:08:02.233 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:248: TEST_recovery_scrub_2: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2024-03-25T23:08:02.233 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: expr 8 - 1 2024-03-25T23:08:02.234 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: seq 0 7 2024-03-25T23:08:02.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-25T23:08:02.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-25T23:08:02.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-25T23:08:02.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:08:02.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-25T23:08:02.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:08:02.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/0 2024-03-25T23:08:02.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4de2c693-e629-4160-940c-ecd79a0b43d9 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-25T23:08:02.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:08:02.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:08:02.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:08:02.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2024-03-25T23:08:02.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2024-03-25T23:08:02.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:08:02.239 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:08:02.239 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-25T23:08:02.239 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:08:02.239 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:08:02.239 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:08:02.239 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:08:02.239 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:08:02.239 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36561 2024-03-25T23:08:02.239 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36561/$cluster-$name.asok' 2024-03-25T23:08:02.239 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36561/$cluster-$name.asok' 2024-03-25T23:08:02.239 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:08:02.239 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:08:02.239 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:08:02.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-25T23:08:02.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-25T23:08:02.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:08:02.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:08:02.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:08:02.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:08:02.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:08:02.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-03-25T23:08:02.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/0 2024-03-25T23:08:02.240 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:08:02.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9c05b965-1fc8-470d-ae5a-32bd12cce1a3 2024-03-25T23:08:02.241 INFO:tasks.workunit.client.0.smithi129.stdout:add osd0 9c05b965-1fc8-470d-ae5a-32bd12cce1a3 2024-03-25T23:08:02.241 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9c05b965-1fc8-470d-ae5a-32bd12cce1a3' 2024-03-25T23:08:02.241 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:08:02.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDSAwJms2FoDxAA/UWVyid0u4ssqhRA3bwyhw== 2024-03-25T23:08:02.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDSAwJms2FoDxAA/UWVyid0u4ssqhRA3bwyhw=="}' 2024-03-25T23:08:02.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9c05b965-1fc8-470d-ae5a-32bd12cce1a3 -i td/osd-recovery-scrub/0/new.json 2024-03-25T23:08:02.512 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:08:02.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/0/new.json 2024-03-25T23:08:02.529 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=4de2c693-e629-4160-940c-ecd79a0b43d9 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36561/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQDSAwJms2FoDxAA/UWVyid0u4ssqhRA3bwyhw== --osd-uuid 9c05b965-1fc8-470d-ae5a-32bd12cce1a3 2024-03-25T23:08:02.563 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:02.564+0000 7f2947c0e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:08:02.580 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:02.580+0000 7f2947c0e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:08:02.581 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:02.580+0000 7f2947c0e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:08:02.582 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:02.580+0000 7f2947c0e7c0 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-03-25T23:08:02.582 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:02.580+0000 7f2947c0e7c0 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-03-25T23:08:02.582 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:02.580+0000 7f2947c0e7c0 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-03-25T23:08:02.680 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:02.680+0000 7f2947c0e7c0 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2024-03-25T23:08:09.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2024-03-25T23:08:09.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:08:09.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-25T23:08:09.034 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd0 key to auth repository 2024-03-25T23:08:09.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:08:09.418 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.0 2024-03-25T23:08:09.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-25T23:08:09.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:08:09.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=4de2c693-e629-4160-940c-ecd79a0b43d9 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36561/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-25T23:08:09.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:08:09.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:08:09.454 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:09.452+0000 7f543baa87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:08:09.486 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:09.484+0000 7f543baa87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:08:09.487 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:09.488+0000 7f543baa87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:08:09.745 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-25T23:08:09.745 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:08:09.745 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T23:08:09.745 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:08:09.745 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:08:09.745 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:08:09.745 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:08:09.746 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:08:09.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:08:09.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:08:10.046 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:10.044+0000 7f543baa87c0 -1 Falling back to public interface 2024-03-25T23:08:10.087 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:08:11.088 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:08:11.088 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:08:11.089 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:08:11.089 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:08:11.089 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:08:11.089 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:08:11.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:08:12.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:08:12.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:08:12.393 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:08:12.393 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:08:12.393 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:08:12.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:08:12.703 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:12.700+0000 7f543baa87c0 -1 osd.0 0 log_to_monitors true 2024-03-25T23:08:12.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:08:13.705 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:08:13.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:08:13.706 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:08:13.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:08:13.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:08:13.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:08:14.022 INFO:tasks.workunit.client.0.smithi129.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/714366827,v1:127.0.0.1:6803/714366827] [v2:127.0.0.1:6804/714366827,v1:127.0.0.1:6805/714366827] exists,up 9c05b965-1fc8-470d-ae5a-32bd12cce1a3 2024-03-25T23:08:14.026 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:08:14.026 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:08:14.026 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:08:14.026 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-25T23:08:14.026 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-25T23:08:14.026 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-25T23:08:14.026 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:08:14.026 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-25T23:08:14.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:08:14.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/1 2024-03-25T23:08:14.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4de2c693-e629-4160-940c-ecd79a0b43d9 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-25T23:08:14.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:08:14.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:08:14.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:08:14.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2024-03-25T23:08:14.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2024-03-25T23:08:14.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:08:14.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:08:14.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-25T23:08:14.027 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:08:14.028 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:08:14.028 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:08:14.028 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:08:14.028 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:08:14.028 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36561 2024-03-25T23:08:14.028 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36561/$cluster-$name.asok' 2024-03-25T23:08:14.028 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36561/$cluster-$name.asok' 2024-03-25T23:08:14.028 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:08:14.028 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:08:14.028 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:08:14.028 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-25T23:08:14.029 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-25T23:08:14.029 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:08:14.029 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:08:14.029 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:08:14.029 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:08:14.029 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:08:14.029 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-03-25T23:08:14.029 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/1 2024-03-25T23:08:14.030 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:08:14.030 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7c66cf98-5b8b-4076-a645-30334d8fb2fd 2024-03-25T23:08:14.030 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 7c66cf98-5b8b-4076-a645-30334d8fb2fd' 2024-03-25T23:08:14.030 INFO:tasks.workunit.client.0.smithi129.stdout:add osd1 7c66cf98-5b8b-4076-a645-30334d8fb2fd 2024-03-25T23:08:14.031 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:08:14.051 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDeAwJmFOwiAxAAmF1LEnNTXuf+cTE7E6C98A== 2024-03-25T23:08:14.051 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDeAwJmFOwiAxAAmF1LEnNTXuf+cTE7E6C98A=="}' 2024-03-25T23:08:14.051 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7c66cf98-5b8b-4076-a645-30334d8fb2fd -i td/osd-recovery-scrub/1/new.json 2024-03-25T23:08:14.472 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:08:14.491 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/1/new.json 2024-03-25T23:08:14.492 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=4de2c693-e629-4160-940c-ecd79a0b43d9 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36561/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQDeAwJmFOwiAxAAmF1LEnNTXuf+cTE7E6C98A== --osd-uuid 7c66cf98-5b8b-4076-a645-30334d8fb2fd 2024-03-25T23:08:14.531 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:14.532+0000 7fc3de46e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:08:14.583 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:14.580+0000 7fc3de46e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:08:14.584 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:14.584+0000 7fc3de46e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:08:14.585 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:14.584+0000 7fc3de46e7c0 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-03-25T23:08:14.585 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:14.584+0000 7fc3de46e7c0 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-03-25T23:08:14.585 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:14.584+0000 7fc3de46e7c0 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-03-25T23:08:14.681 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:14.680+0000 7fc3de46e7c0 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2024-03-25T23:08:21.436 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2024-03-25T23:08:21.437 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:08:21.438 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd1 key to auth repository 2024-03-25T23:08:21.438 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-25T23:08:21.438 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:08:21.824 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.1 2024-03-25T23:08:21.825 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-25T23:08:21.825 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:08:21.825 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=4de2c693-e629-4160-940c-ecd79a0b43d9 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36561/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-25T23:08:21.825 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:08:21.825 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:08:21.861 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:21.860+0000 7ff6b79947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:08:21.895 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:21.896+0000 7ff6b79947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:08:21.897 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:21.896+0000 7ff6b79947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:08:22.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-25T23:08:22.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:08:22.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T23:08:22.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:08:22.161 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:08:22.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:08:22.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:08:22.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:08:22.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:08:22.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:08:22.450 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:22.448+0000 7ff6b79947c0 -1 Falling back to public interface 2024-03-25T23:08:22.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:08:23.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:08:23.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:08:23.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:08:23.511 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:08:23.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:08:23.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:08:23.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:08:24.815 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:08:24.816 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:08:24.816 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:08:24.816 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:08:24.816 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:08:24.816 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:08:25.134 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:08:25.172 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:25.172+0000 7ff6b79947c0 -1 osd.1 0 log_to_monitors true 2024-03-25T23:08:26.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:08:26.137 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:08:26.137 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:08:26.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:08:26.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:08:26.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:08:26.508 INFO:tasks.workunit.client.0.smithi129.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2506236697,v1:127.0.0.1:6811/2506236697] [v2:127.0.0.1:6812/2506236697,v1:127.0.0.1:6813/2506236697] exists,up 7c66cf98-5b8b-4076-a645-30334d8fb2fd 2024-03-25T23:08:26.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:08:26.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:08:26.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:08:26.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-25T23:08:26.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-25T23:08:26.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-25T23:08:26.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:08:26.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-25T23:08:26.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:08:26.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/2 2024-03-25T23:08:26.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4de2c693-e629-4160-940c-ecd79a0b43d9 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-25T23:08:26.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:08:26.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:08:26.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:08:26.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2024-03-25T23:08:26.512 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2024-03-25T23:08:26.512 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:08:26.512 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:08:26.512 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-25T23:08:26.512 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:08:26.512 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:08:26.512 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:08:26.512 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:08:26.512 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:08:26.512 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36561 2024-03-25T23:08:26.512 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36561/$cluster-$name.asok' 2024-03-25T23:08:26.512 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36561/$cluster-$name.asok' 2024-03-25T23:08:26.512 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:08:26.512 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:08:26.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:08:26.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-25T23:08:26.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-25T23:08:26.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:08:26.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:08:26.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:08:26.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:08:26.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:08:26.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-03-25T23:08:26.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/2 2024-03-25T23:08:26.514 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:08:26.514 INFO:tasks.workunit.client.0.smithi129.stdout:add osd2 9e8fe362-4a2a-4b7f-a2ea-09dfb2480ce0 2024-03-25T23:08:26.514 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9e8fe362-4a2a-4b7f-a2ea-09dfb2480ce0 2024-03-25T23:08:26.514 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 9e8fe362-4a2a-4b7f-a2ea-09dfb2480ce0' 2024-03-25T23:08:26.514 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:08:26.534 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDqAwJm9+P1HxAAxHarN979LbgBJLgQd1duUA== 2024-03-25T23:08:26.534 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDqAwJm9+P1HxAAxHarN979LbgBJLgQd1duUA=="}' 2024-03-25T23:08:26.535 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9e8fe362-4a2a-4b7f-a2ea-09dfb2480ce0 -i td/osd-recovery-scrub/2/new.json 2024-03-25T23:08:26.924 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:08:26.941 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/2/new.json 2024-03-25T23:08:26.942 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=4de2c693-e629-4160-940c-ecd79a0b43d9 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36561/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQDqAwJm9+P1HxAAxHarN979LbgBJLgQd1duUA== --osd-uuid 9e8fe362-4a2a-4b7f-a2ea-09dfb2480ce0 2024-03-25T23:08:26.981 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:26.980+0000 7fbf768117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:08:27.008 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:27.008+0000 7fbf768117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:08:27.009 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:27.008+0000 7fbf768117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:08:27.010 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:27.008+0000 7fbf768117c0 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-03-25T23:08:27.010 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:27.008+0000 7fbf768117c0 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-03-25T23:08:27.010 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:27.008+0000 7fbf768117c0 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-03-25T23:08:27.079 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:27.076+0000 7fbf768117c0 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2024-03-25T23:08:34.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2024-03-25T23:08:34.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:08:34.389 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd2 key to auth repository 2024-03-25T23:08:34.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-25T23:08:34.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:08:34.765 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.2 2024-03-25T23:08:34.766 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-25T23:08:34.766 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:08:34.766 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=4de2c693-e629-4160-940c-ecd79a0b43d9 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36561/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-25T23:08:34.766 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:08:34.766 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:08:34.801 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:34.800+0000 7fe82fee87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:08:34.803 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:34.804+0000 7fe82fee87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:08:34.805 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:34.804+0000 7fe82fee87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:08:35.105 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-25T23:08:35.105 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:08:35.105 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-25T23:08:35.105 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:08:35.105 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:08:35.105 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:08:35.106 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:08:35.106 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:08:35.106 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:08:35.106 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:08:35.341 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:35.340+0000 7fe82fee87c0 -1 Falling back to public interface 2024-03-25T23:08:35.426 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:08:36.428 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:08:36.428 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:08:36.428 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:08:36.428 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:08:36.429 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:08:36.429 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:08:36.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:08:37.754 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:08:37.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:08:37.755 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:08:37.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:08:37.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:08:37.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:08:37.998 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:37.996+0000 7fe82fee87c0 -1 osd.2 0 log_to_monitors true 2024-03-25T23:08:38.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:08:39.134 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:08:39.134 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:08:39.134 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:08:39.134 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:08:39.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:08:39.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:08:39.452 INFO:tasks.workunit.client.0.smithi129.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/283379661,v1:127.0.0.1:6819/283379661] [v2:127.0.0.1:6820/283379661,v1:127.0.0.1:6821/283379661] exists,up 9e8fe362-4a2a-4b7f-a2ea-09dfb2480ce0 2024-03-25T23:08:39.455 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:08:39.456 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:08:39.456 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:08:39.456 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-25T23:08:39.456 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-25T23:08:39.456 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-25T23:08:39.456 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:08:39.456 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-03-25T23:08:39.456 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:08:39.456 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/3 2024-03-25T23:08:39.456 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4de2c693-e629-4160-940c-ecd79a0b43d9 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-25T23:08:39.456 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:08:39.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:08:39.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:08:39.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2024-03-25T23:08:39.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2024-03-25T23:08:39.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:08:39.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:08:39.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-25T23:08:39.457 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:08:39.457 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:08:39.457 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:08:39.457 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:08:39.457 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:08:39.457 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36561 2024-03-25T23:08:39.457 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36561/$cluster-$name.asok' 2024-03-25T23:08:39.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36561/$cluster-$name.asok' 2024-03-25T23:08:39.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:08:39.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:08:39.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:08:39.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-25T23:08:39.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-25T23:08:39.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:08:39.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:08:39.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:08:39.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:08:39.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:08:39.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-03-25T23:08:39.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/3 2024-03-25T23:08:39.459 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:08:39.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=018ef372-2c3f-4a6d-98ce-44d01b864903 2024-03-25T23:08:39.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 018ef372-2c3f-4a6d-98ce-44d01b864903' 2024-03-25T23:08:39.459 INFO:tasks.workunit.client.0.smithi129.stdout:add osd3 018ef372-2c3f-4a6d-98ce-44d01b864903 2024-03-25T23:08:39.460 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:08:39.479 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD3AwJmz9WuHBAAU0zPk+QJ6sgex8oYk6n/1w== 2024-03-25T23:08:39.479 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD3AwJmz9WuHBAAU0zPk+QJ6sgex8oYk6n/1w=="}' 2024-03-25T23:08:39.479 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 018ef372-2c3f-4a6d-98ce-44d01b864903 -i td/osd-recovery-scrub/3/new.json 2024-03-25T23:08:39.862 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:08:39.878 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/3/new.json 2024-03-25T23:08:39.879 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=4de2c693-e629-4160-940c-ecd79a0b43d9 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36561/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQD3AwJmz9WuHBAAU0zPk+QJ6sgex8oYk6n/1w== --osd-uuid 018ef372-2c3f-4a6d-98ce-44d01b864903 2024-03-25T23:08:39.912 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:39.912+0000 7faab579c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:08:39.963 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:39.964+0000 7faab579c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:08:39.965 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:39.964+0000 7faab579c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:08:39.965 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:39.964+0000 7faab579c7c0 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-03-25T23:08:39.965 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:39.964+0000 7faab579c7c0 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-03-25T23:08:39.965 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:39.964+0000 7faab579c7c0 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-03-25T23:08:40.047 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:40.048+0000 7faab579c7c0 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2024-03-25T23:08:46.942 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2024-03-25T23:08:46.942 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:08:46.943 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd3 key to auth repository 2024-03-25T23:08:46.943 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-03-25T23:08:46.943 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:08:47.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-03-25T23:08:47.345 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.3 2024-03-25T23:08:47.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:08:47.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=4de2c693-e629-4160-940c-ecd79a0b43d9 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36561/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-25T23:08:47.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:08:47.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:08:47.381 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:47.379+0000 7f9dc37617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:08:47.406 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:47.403+0000 7f9dc37617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:08:47.407 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:47.407+0000 7f9dc37617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:08:47.643 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-03-25T23:08:47.643 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:08:47.643 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-25T23:08:47.643 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:08:47.643 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:08:47.643 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:08:47.644 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:08:47.644 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:08:47.644 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:08:47.644 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:08:47.938 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:08:47.956 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:47.955+0000 7f9dc37617c0 -1 Falling back to public interface 2024-03-25T23:08:48.940 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:08:48.940 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:08:48.941 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:08:48.941 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:08:48.941 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:08:48.942 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:08:49.249 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:08:50.251 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:08:50.251 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:08:50.252 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:08:50.253 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:08:50.253 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:08:50.253 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:08:50.515 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:50.515+0000 7f9dc37617c0 -1 osd.3 0 log_to_monitors true 2024-03-25T23:08:50.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:08:51.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:08:51.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:08:51.562 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:08:51.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:08:51.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:08:51.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:08:51.892 INFO:tasks.workunit.client.0.smithi129.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1080182781,v1:127.0.0.1:6827/1080182781] [v2:127.0.0.1:6828/1080182781,v1:127.0.0.1:6829/1080182781] exists,up 018ef372-2c3f-4a6d-98ce-44d01b864903 2024-03-25T23:08:51.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:08:51.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:08:51.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:08:51.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-25T23:08:51.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 4 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-25T23:08:51.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-25T23:08:51.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:08:51.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-03-25T23:08:51.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:08:51.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/4 2024-03-25T23:08:51.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4de2c693-e629-4160-940c-ecd79a0b43d9 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-25T23:08:51.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:08:51.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:08:51.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:08:51.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/4' 2024-03-25T23:08:51.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/4/journal' 2024-03-25T23:08:51.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:08:51.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:08:51.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-25T23:08:51.897 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:08:51.898 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:08:51.898 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:08:51.898 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:08:51.898 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:08:51.898 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36561 2024-03-25T23:08:51.898 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36561/$cluster-$name.asok' 2024-03-25T23:08:51.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36561/$cluster-$name.asok' 2024-03-25T23:08:51.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:08:51.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:08:51.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:08:51.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-25T23:08:51.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-25T23:08:51.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:08:51.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:08:51.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:08:51.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:08:51.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:08:51.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-03-25T23:08:51.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/4 2024-03-25T23:08:51.899 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:08:51.900 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b12cb75c-f02b-4496-9588-4c1d5e36a8a0 2024-03-25T23:08:51.900 INFO:tasks.workunit.client.0.smithi129.stdout:add osd4 b12cb75c-f02b-4496-9588-4c1d5e36a8a0 2024-03-25T23:08:51.900 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 b12cb75c-f02b-4496-9588-4c1d5e36a8a0' 2024-03-25T23:08:51.900 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:08:51.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQADBAJm8lfzNhAAdE+2x+PfptXunUdBGX6aaA== 2024-03-25T23:08:51.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQADBAJm8lfzNhAAdE+2x+PfptXunUdBGX6aaA=="}' 2024-03-25T23:08:51.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b12cb75c-f02b-4496-9588-4c1d5e36a8a0 -i td/osd-recovery-scrub/4/new.json 2024-03-25T23:08:52.349 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:08:52.366 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/4/new.json 2024-03-25T23:08:52.367 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=4de2c693-e629-4160-940c-ecd79a0b43d9 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36561/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQADBAJm8lfzNhAAdE+2x+PfptXunUdBGX6aaA== --osd-uuid b12cb75c-f02b-4496-9588-4c1d5e36a8a0 2024-03-25T23:08:52.399 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:52.399+0000 7f24d56ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:08:52.468 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:52.467+0000 7f24d56ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:08:52.470 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:52.467+0000 7f24d56ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:08:52.470 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:52.467+0000 7f24d56ab7c0 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-03-25T23:08:52.470 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:52.467+0000 7f24d56ab7c0 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-03-25T23:08:52.470 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:52.467+0000 7f24d56ab7c0 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-03-25T23:08:52.592 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:08:52.591+0000 7f24d56ab7c0 -1 bluestore(td/osd-recovery-scrub/4) _read_fsid unparsable uuid 2024-03-25T23:08:59.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/4/keyring 2024-03-25T23:08:59.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:08:59.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-03-25T23:08:59.833 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd4 key to auth repository 2024-03-25T23:08:59.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:09:00.213 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.4 2024-03-25T23:09:00.214 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-03-25T23:09:00.214 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=4de2c693-e629-4160-940c-ecd79a0b43d9 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36561/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-25T23:09:00.214 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:09:00.214 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:09:00.214 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:09:00.249 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:09:00.247+0000 7f64675137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:09:00.285 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:09:00.283+0000 7f64675137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:09:00.286 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:09:00.287+0000 7f64675137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:09:00.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-03-25T23:09:00.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:09:00.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-03-25T23:09:00.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:09:00.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:09:00.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:09:00.542 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:09:00.542 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:09:00.542 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:09:00.542 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:09:00.841 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:09:00.839+0000 7f64675137c0 -1 Falling back to public interface 2024-03-25T23:09:00.885 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:09:01.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:09:01.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:09:01.887 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:09:01.888 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:09:01.888 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:09:01.888 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:09:02.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:09:03.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:09:03.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:09:03.200 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:09:03.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:09:03.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:09:03.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:09:03.556 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:09:03.581 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:09:03.579+0000 7f64675137c0 -1 osd.4 0 log_to_monitors true 2024-03-25T23:09:04.557 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:09:04.557 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:09:04.558 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:09:04.558 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:09:04.558 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:09:04.559 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:09:05.007 INFO:tasks.workunit.client.0.smithi129.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2620064272,v1:127.0.0.1:6835/2620064272] [v2:127.0.0.1:6836/2620064272,v1:127.0.0.1:6837/2620064272] exists,up b12cb75c-f02b-4496-9588-4c1d5e36a8a0 2024-03-25T23:09:05.009 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:09:05.009 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:09:05.009 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:09:05.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-25T23:09:05.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 5 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-25T23:09:05.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-25T23:09:05.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:09:05.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-03-25T23:09:05.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:09:05.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/5 2024-03-25T23:09:05.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4de2c693-e629-4160-940c-ecd79a0b43d9 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-25T23:09:05.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:09:05.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:09:05.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:09:05.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/5' 2024-03-25T23:09:05.011 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/5/journal' 2024-03-25T23:09:05.011 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:09:05.011 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:09:05.011 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-25T23:09:05.011 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:09:05.011 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:09:05.011 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:09:05.011 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:09:05.011 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:09:05.011 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36561 2024-03-25T23:09:05.012 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36561/$cluster-$name.asok' 2024-03-25T23:09:05.012 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36561/$cluster-$name.asok' 2024-03-25T23:09:05.012 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:09:05.012 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:09:05.013 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:09:05.013 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-25T23:09:05.013 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-25T23:09:05.013 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:09:05.013 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:09:05.013 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:09:05.013 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:09:05.013 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:09:05.013 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-03-25T23:09:05.013 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/5 2024-03-25T23:09:05.013 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:09:05.014 INFO:tasks.workunit.client.0.smithi129.stdout:add osd5 01444aa6-35ed-4b75-ba7a-e6883931bf58 2024-03-25T23:09:05.014 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=01444aa6-35ed-4b75-ba7a-e6883931bf58 2024-03-25T23:09:05.014 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 01444aa6-35ed-4b75-ba7a-e6883931bf58' 2024-03-25T23:09:05.014 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:09:05.035 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQARBAJm0LA+AhAAbh4Nt/D0Zo9xfWUupGPeVw== 2024-03-25T23:09:05.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQARBAJm0LA+AhAAbh4Nt/D0Zo9xfWUupGPeVw=="}' 2024-03-25T23:09:05.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 01444aa6-35ed-4b75-ba7a-e6883931bf58 -i td/osd-recovery-scrub/5/new.json 2024-03-25T23:09:05.474 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:09:05.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/5/new.json 2024-03-25T23:09:05.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=4de2c693-e629-4160-940c-ecd79a0b43d9 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36561/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQARBAJm0LA+AhAAbh4Nt/D0Zo9xfWUupGPeVw== --osd-uuid 01444aa6-35ed-4b75-ba7a-e6883931bf58 2024-03-25T23:09:05.528 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:09:05.527+0000 7fd78057a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:09:05.574 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:09:05.571+0000 7fd78057a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:09:05.575 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:09:05.575+0000 7fd78057a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:09:05.576 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:09:05.575+0000 7fd78057a7c0 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-03-25T23:09:05.576 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:09:05.575+0000 7fd78057a7c0 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-03-25T23:09:05.576 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:09:05.575+0000 7fd78057a7c0 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-03-25T23:09:05.660 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:09:05.659+0000 7fd78057a7c0 -1 bluestore(td/osd-recovery-scrub/5) _read_fsid unparsable uuid 2024-03-25T23:09:12.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/5/keyring 2024-03-25T23:09:12.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:09:12.973 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd5 key to auth repository 2024-03-25T23:09:12.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-03-25T23:09:12.974 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:09:13.369 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.5 2024-03-25T23:09:13.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-03-25T23:09:13.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=4de2c693-e629-4160-940c-ecd79a0b43d9 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36561/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-25T23:09:13.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:09:13.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:09:13.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:09:13.405 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:09:13.403+0000 7f3da70be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:09:13.411 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:09:13.411+0000 7f3da70be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:09:13.412 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:09:13.411+0000 7f3da70be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:09:13.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-03-25T23:09:13.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:09:13.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-25T23:09:13.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:09:13.704 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:09:13.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:09:13.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:09:13.705 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:09:13.705 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:09:13.705 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:09:13.953 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:09:13.951+0000 7f3da70be7c0 -1 Falling back to public interface 2024-03-25T23:09:14.054 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:09:15.055 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:09:15.055 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:09:15.056 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:09:15.056 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:09:15.056 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:09:15.056 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:09:15.417 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:09:16.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:09:16.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:09:16.419 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:09:16.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:09:16.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:09:16.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:09:16.560 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:09:16.559+0000 7f3da70be7c0 -1 osd.5 0 log_to_monitors true 2024-03-25T23:09:16.821 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:09:17.822 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:09:17.822 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:09:17.823 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:09:17.823 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:09:17.823 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:09:17.823 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:09:18.132 INFO:tasks.workunit.client.0.smithi129.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/1762900183,v1:127.0.0.1:6843/1762900183] [v2:127.0.0.1:6844/1762900183,v1:127.0.0.1:6845/1762900183] exists,up 01444aa6-35ed-4b75-ba7a-e6883931bf58 2024-03-25T23:09:18.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:09:18.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:09:18.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:09:18.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-25T23:09:18.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 6 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-25T23:09:18.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-25T23:09:18.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:09:18.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-03-25T23:09:18.137 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:09:18.137 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/6 2024-03-25T23:09:18.137 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4de2c693-e629-4160-940c-ecd79a0b43d9 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-25T23:09:18.137 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:09:18.137 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:09:18.137 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:09:18.137 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/6' 2024-03-25T23:09:18.137 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/6/journal' 2024-03-25T23:09:18.137 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:09:18.137 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:09:18.137 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-25T23:09:18.137 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:09:18.138 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:09:18.138 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:09:18.138 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:09:18.138 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:09:18.138 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36561 2024-03-25T23:09:18.138 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36561/$cluster-$name.asok' 2024-03-25T23:09:18.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36561/$cluster-$name.asok' 2024-03-25T23:09:18.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:09:18.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:09:18.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:09:18.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-25T23:09:18.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-25T23:09:18.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:09:18.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:09:18.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:09:18.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:09:18.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:09:18.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-03-25T23:09:18.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/6 2024-03-25T23:09:18.139 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:09:18.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=264205ba-25c6-4b1b-ab70-9efa3c4c06c4 2024-03-25T23:09:18.140 INFO:tasks.workunit.client.0.smithi129.stdout:add osd6 264205ba-25c6-4b1b-ab70-9efa3c4c06c4 2024-03-25T23:09:18.141 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 264205ba-25c6-4b1b-ab70-9efa3c4c06c4' 2024-03-25T23:09:18.141 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:09:18.158 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAeBAJmVBWaCRAAqwPkrkRjZJpEAfFeOR29Iw== 2024-03-25T23:09:18.158 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAeBAJmVBWaCRAAqwPkrkRjZJpEAfFeOR29Iw=="}' 2024-03-25T23:09:18.158 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 264205ba-25c6-4b1b-ab70-9efa3c4c06c4 -i td/osd-recovery-scrub/6/new.json 2024-03-25T23:09:18.600 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:09:18.619 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/6/new.json 2024-03-25T23:09:18.620 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=4de2c693-e629-4160-940c-ecd79a0b43d9 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36561/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQAeBAJmVBWaCRAAqwPkrkRjZJpEAfFeOR29Iw== --osd-uuid 264205ba-25c6-4b1b-ab70-9efa3c4c06c4 2024-03-25T23:09:18.659 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:09:18.659+0000 7f87087af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:09:18.711 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:09:18.711+0000 7f87087af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:09:18.713 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:09:18.711+0000 7f87087af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:09:18.713 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:09:18.711+0000 7f87087af7c0 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-03-25T23:09:18.714 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:09:18.711+0000 7f87087af7c0 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-03-25T23:09:18.714 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:09:18.711+0000 7f87087af7c0 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-03-25T23:09:18.788 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:09:18.787+0000 7f87087af7c0 -1 bluestore(td/osd-recovery-scrub/6) _read_fsid unparsable uuid 2024-03-25T23:09:25.788 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/6/keyring 2024-03-25T23:09:25.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:09:25.790 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-03-25T23:09:25.791 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd6 key to auth repository 2024-03-25T23:09:25.791 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:09:26.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-03-25T23:09:26.170 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.6 2024-03-25T23:09:26.170 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:09:26.170 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:09:26.171 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=4de2c693-e629-4160-940c-ecd79a0b43d9 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36561/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-25T23:09:26.171 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:09:26.207 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:09:26.207+0000 7f2d6f9727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:09:26.244 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:09:26.243+0000 7f2d6f9727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:09:26.246 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:09:26.243+0000 7f2d6f9727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:09:26.526 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-03-25T23:09:26.526 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:09:26.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-03-25T23:09:26.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:09:26.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:09:26.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:09:26.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:09:26.527 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:09:26.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:09:26.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:09:26.809 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:09:26.807+0000 7f2d6f9727c0 -1 Falling back to public interface 2024-03-25T23:09:26.876 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:09:27.877 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:09:27.877 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:09:27.877 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:09:27.878 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:09:27.878 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:09:27.878 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:09:28.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:09:29.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:09:29.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:09:29.189 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:09:29.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:09:29.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:09:29.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:09:29.547 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:09:29.547 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:09:29.547+0000 7f2d6f9727c0 -1 osd.6 0 log_to_monitors true 2024-03-25T23:09:30.548 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:09:30.548 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:09:30.548 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:09:30.549 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:09:30.549 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:09:30.549 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:09:30.959 INFO:tasks.workunit.client.0.smithi129.stdout:osd.6 up in weight 1 up_from 34 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/3040435919,v1:127.0.0.1:6851/3040435919] [v2:127.0.0.1:6852/3040435919,v1:127.0.0.1:6853/3040435919] exists,up 264205ba-25c6-4b1b-ab70-9efa3c4c06c4 2024-03-25T23:09:30.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:09:30.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:09:30.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:09:30.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-25T23:09:30.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 7 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-25T23:09:30.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-25T23:09:30.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:09:30.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-03-25T23:09:30.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:09:30.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/7 2024-03-25T23:09:30.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4de2c693-e629-4160-940c-ecd79a0b43d9 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-25T23:09:30.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:09:30.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:09:30.965 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:09:30.965 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/7' 2024-03-25T23:09:30.965 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/7/journal' 2024-03-25T23:09:30.965 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:09:30.965 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:09:30.965 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-25T23:09:30.965 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:09:30.965 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:09:30.965 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:09:30.965 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:09:30.965 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:09:30.965 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36561 2024-03-25T23:09:30.965 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36561/$cluster-$name.asok' 2024-03-25T23:09:30.965 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36561/$cluster-$name.asok' 2024-03-25T23:09:30.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:09:30.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:09:30.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:09:30.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-25T23:09:30.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-25T23:09:30.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:09:30.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:09:30.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:09:30.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:09:30.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:09:30.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-03-25T23:09:30.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/7 2024-03-25T23:09:30.967 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:09:30.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ce814055-ccb6-479b-b497-84501553a4a4 2024-03-25T23:09:30.967 INFO:tasks.workunit.client.0.smithi129.stdout:add osd7 ce814055-ccb6-479b-b497-84501553a4a4 2024-03-25T23:09:30.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 ce814055-ccb6-479b-b497-84501553a4a4' 2024-03-25T23:09:30.968 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:09:30.982 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAqBAJmJq+wOhAAFXLbs7sX2WmMydQ/LfvTWw== 2024-03-25T23:09:30.982 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAqBAJmJq+wOhAAFXLbs7sX2WmMydQ/LfvTWw=="}' 2024-03-25T23:09:30.982 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ce814055-ccb6-479b-b497-84501553a4a4 -i td/osd-recovery-scrub/7/new.json 2024-03-25T23:09:31.452 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:09:31.470 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/7/new.json 2024-03-25T23:09:31.471 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=4de2c693-e629-4160-940c-ecd79a0b43d9 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36561/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQAqBAJmJq+wOhAAFXLbs7sX2WmMydQ/LfvTWw== --osd-uuid ce814055-ccb6-479b-b497-84501553a4a4 2024-03-25T23:09:31.509 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:09:31.507+0000 7f5a08af87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:09:31.544 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:09:31.543+0000 7f5a08af87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:09:31.546 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:09:31.547+0000 7f5a08af87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:09:31.547 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:09:31.547+0000 7f5a08af87c0 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-03-25T23:09:31.547 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:09:31.547+0000 7f5a08af87c0 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-03-25T23:09:31.547 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:09:31.547+0000 7f5a08af87c0 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-03-25T23:09:31.646 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:09:31.647+0000 7f5a08af87c0 -1 bluestore(td/osd-recovery-scrub/7) _read_fsid unparsable uuid 2024-03-25T23:09:39.003 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/7/keyring 2024-03-25T23:09:39.004 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:09:39.005 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd7 key to auth repository 2024-03-25T23:09:39.005 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-03-25T23:09:39.005 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:09:39.434 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.7 2024-03-25T23:09:39.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-03-25T23:09:39.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:09:39.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=4de2c693-e629-4160-940c-ecd79a0b43d9 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36561/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-25T23:09:39.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:09:39.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:09:39.472 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:09:39.471+0000 7fb6a85bb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:09:39.507 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:09:39.507+0000 7fb6a85bb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:09:39.508 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:09:39.507+0000 7fb6a85bb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:09:39.790 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-03-25T23:09:39.791 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:09:39.791 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-03-25T23:09:39.791 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:09:39.791 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:09:39.791 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:09:39.791 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:09:39.792 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:09:39.792 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:09:39.792 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:09:40.064 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:09:40.063+0000 7fb6a85bb7c0 -1 Falling back to public interface 2024-03-25T23:09:40.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:09:41.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:09:41.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:09:41.143 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:09:41.143 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:09:41.143 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:09:41.143 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:09:41.463 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:09:42.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:09:42.465 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:09:42.465 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:09:42.465 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:09:42.466 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:09:42.466 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:09:42.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:09:42.895 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:09:42.895+0000 7fb6a85bb7c0 -1 osd.7 0 log_to_monitors true 2024-03-25T23:09:43.785 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:09:43.785 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:09:43.785 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:09:43.786 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:09:43.786 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:09:43.786 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:09:44.103 INFO:tasks.workunit.client.0.smithi129.stdout:osd.7 up in weight 1 up_from 39 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/546675164,v1:127.0.0.1:6859/546675164] [v2:127.0.0.1:6860/546675164,v1:127.0.0.1:6861/546675164] exists,up ce814055-ccb6-479b-b497-84501553a4a4 2024-03-25T23:09:44.104 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:09:44.104 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:09:44.104 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:09:44.104 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:256: TEST_recovery_scrub_2: create_pool test 32 32 2024-03-25T23:09:44.104 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 32 32 2024-03-25T23:09:44.749 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'test' created 2024-03-25T23:09:44.768 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T23:09:45.771 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:257: TEST_recovery_scrub_2: wait_for_clean 2024-03-25T23:09:45.771 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:09:45.771 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:09:45.771 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:09:45.771 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:09:45.772 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:09:45.772 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:09:45.773 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:09:45.773 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:09:45.773 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:09:45.893 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:09:45.893 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:09:45.893 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:09:45.894 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:09:45.894 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:09:45.894 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:09:46.545 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:09:46.545 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:09:46.545 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:09:46.545 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:09:46.545 INFO:tasks.workunit.client.0.smithi129.stderr:4 2024-03-25T23:09:46.545 INFO:tasks.workunit.client.0.smithi129.stderr:5 2024-03-25T23:09:46.545 INFO:tasks.workunit.client.0.smithi129.stderr:6 2024-03-25T23:09:46.545 INFO:tasks.workunit.client.0.smithi129.stderr:7' 2024-03-25T23:09:46.545 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:09:46.545 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:09:46.546 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:09:46.841 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836500 2024-03-25T23:09:46.841 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836500 2024-03-25T23:09:46.841 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500' 2024-03-25T23:09:46.841 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:09:46.841 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:09:47.080 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672979 2024-03-25T23:09:47.080 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672979 2024-03-25T23:09:47.080 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672979' 2024-03-25T23:09:47.080 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:09:47.080 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:09:47.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509455 2024-03-25T23:09:47.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509455 2024-03-25T23:09:47.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672979 2-64424509455' 2024-03-25T23:09:47.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:09:47.348 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:09:47.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345933 2024-03-25T23:09:47.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345933 2024-03-25T23:09:47.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672979 2-64424509455 3-85899345933' 2024-03-25T23:09:47.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:09:47.579 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T23:09:47.849 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182410 2024-03-25T23:09:47.849 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182410 2024-03-25T23:09:47.849 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672979 2-64424509455 3-85899345933 4-107374182410' 2024-03-25T23:09:47.849 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:09:47.849 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T23:09:47.995 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018887 2024-03-25T23:09:47.995 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018887 2024-03-25T23:09:47.995 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672979 2-64424509455 3-85899345933 4-107374182410 5-128849018887' 2024-03-25T23:09:47.996 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:09:47.996 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T23:09:48.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888069 2024-03-25T23:09:48.137 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888069 2024-03-25T23:09:48.137 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672979 2-64424509455 3-85899345933 4-107374182410 5-128849018887 6-146028888069' 2024-03-25T23:09:48.137 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:09:48.137 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-25T23:09:48.286 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724546 2024-03-25T23:09:48.286 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724546 2024-03-25T23:09:48.286 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672979 2-64424509455 3-85899345933 4-107374182410 5-128849018887 6-146028888069 7-167503724546' 2024-03-25T23:09:48.286 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:09:48.287 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836500 2024-03-25T23:09:48.287 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:09:48.287 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:09:48.288 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836500 2024-03-25T23:09:48.288 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:09:48.289 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836500 2024-03-25T23:09:48.289 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 21474836500 2024-03-25T23:09:48.290 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-03-25T23:09:48.290 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:09:48.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836500 2024-03-25T23:09:48.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:09:49.610 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:09:49.611 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:09:49.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836500 2024-03-25T23:09:49.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:09:49.927 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:09:49.927 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672979 2024-03-25T23:09:49.928 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:09:49.929 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672979 2024-03-25T23:09:49.929 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:09:49.931 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672979 2024-03-25T23:09:49.931 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-03-25T23:09:49.931 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 42949672979 2024-03-25T23:09:49.931 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:09:50.244 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672979 -lt 42949672979 2024-03-25T23:09:50.244 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:09:50.245 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509455 2024-03-25T23:09:50.245 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:09:50.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:09:50.247 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509455 2024-03-25T23:09:50.247 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:09:50.249 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509455 2024-03-25T23:09:50.249 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-03-25T23:09:50.249 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 64424509455 2024-03-25T23:09:50.249 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:09:50.574 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509455 -lt 64424509455 2024-03-25T23:09:50.574 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:09:50.574 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345933 2024-03-25T23:09:50.574 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:09:50.576 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:09:50.577 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345933 2024-03-25T23:09:50.577 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:09:50.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345933 2024-03-25T23:09:50.578 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.3 seq 85899345933 2024-03-25T23:09:50.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345933' 2024-03-25T23:09:50.579 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:09:50.935 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345933 -lt 85899345933 2024-03-25T23:09:50.935 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:09:50.935 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182410 2024-03-25T23:09:50.936 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:09:50.937 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T23:09:50.938 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:09:50.938 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182410 2024-03-25T23:09:50.939 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182410 2024-03-25T23:09:50.939 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.4 seq 107374182410 2024-03-25T23:09:50.939 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182410' 2024-03-25T23:09:50.940 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T23:09:51.250 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182411 -lt 107374182410 2024-03-25T23:09:51.250 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:09:51.251 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018887 2024-03-25T23:09:51.251 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:09:51.252 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T23:09:51.253 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018887 2024-03-25T23:09:51.253 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:09:51.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018887 2024-03-25T23:09:51.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018887' 2024-03-25T23:09:51.255 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.5 seq 128849018887 2024-03-25T23:09:51.255 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T23:09:51.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018888 -lt 128849018887 2024-03-25T23:09:51.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:09:51.582 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-146028888069 2024-03-25T23:09:51.582 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:09:51.583 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-25T23:09:51.584 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-146028888069 2024-03-25T23:09:51.584 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:09:51.585 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.6 seq 146028888069 2024-03-25T23:09:51.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888069 2024-03-25T23:09:51.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 146028888069' 2024-03-25T23:09:51.585 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-25T23:09:51.924 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888069 -lt 146028888069 2024-03-25T23:09:51.924 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:09:51.925 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-167503724546 2024-03-25T23:09:51.925 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:09:51.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-25T23:09:51.927 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-167503724546 2024-03-25T23:09:51.927 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:09:51.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724546 2024-03-25T23:09:51.928 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.7 seq 167503724546 2024-03-25T23:09:51.928 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 167503724546' 2024-03-25T23:09:51.928 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-25T23:09:52.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724547 -lt 167503724546 2024-03-25T23:09:52.262 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:09:52.262 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:09:52.262 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:09:52.675 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-03-25T23:09:52.675 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:09:52.676 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:09:52.676 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:09:52.676 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:09:52.676 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:09:52.676 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:09:52.676 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:09:52.969 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-03-25T23:09:52.969 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:09:52.969 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:09:52.970 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:09:53.311 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-03-25T23:09:53.311 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:09:53.311 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:09:53.311 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: ceph osd dump 2024-03-25T23:09:53.311 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: grep '^pool.*['\'']test['\'']' 2024-03-25T23:09:53.311 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: awk '{ print $2 }' 2024-03-25T23:09:53.606 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: poolid=1 2024-03-25T23:09:53.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:260: TEST_recovery_scrub_2: dd if=/dev/urandom of=testdata.36561 bs=1M count=50 2024-03-25T23:09:53.775 INFO:tasks.workunit.client.0.smithi129.stderr:50+0 records in 2024-03-25T23:09:53.775 INFO:tasks.workunit.client.0.smithi129.stderr:50+0 records out 2024-03-25T23:09:53.775 INFO:tasks.workunit.client.0.smithi129.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.168021 s, 312 MB/s 2024-03-25T23:09:53.776 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: seq 1 40 2024-03-25T23:09:53.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-25T23:09:53.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj1 testdata.36561 2024-03-25T23:09:55.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-25T23:09:55.647 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj2 testdata.36561 2024-03-25T23:09:57.455 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-25T23:09:57.455 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj3 testdata.36561 2024-03-25T23:09:59.251 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-25T23:09:59.251 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj4 testdata.36561 2024-03-25T23:10:01.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-25T23:10:01.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj5 testdata.36561 2024-03-25T23:10:02.882 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-25T23:10:02.882 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj6 testdata.36561 2024-03-25T23:10:04.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-25T23:10:04.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj7 testdata.36561 2024-03-25T23:10:06.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-25T23:10:06.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj8 testdata.36561 2024-03-25T23:10:08.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-25T23:10:08.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj9 testdata.36561 2024-03-25T23:10:10.451 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-25T23:10:10.451 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj10 testdata.36561 2024-03-25T23:10:12.308 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-25T23:10:12.308 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj11 testdata.36561 2024-03-25T23:10:14.233 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-25T23:10:14.233 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj12 testdata.36561 2024-03-25T23:10:16.065 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-25T23:10:16.065 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj13 testdata.36561 2024-03-25T23:10:17.938 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-25T23:10:17.939 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj14 testdata.36561 2024-03-25T23:10:19.821 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-25T23:10:19.821 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj15 testdata.36561 2024-03-25T23:10:21.628 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-25T23:10:21.628 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj16 testdata.36561 2024-03-25T23:10:23.327 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-25T23:10:23.362 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj17 testdata.36561 2024-03-25T23:10:25.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-25T23:10:25.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj18 testdata.36561 2024-03-25T23:10:26.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-25T23:10:26.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj19 testdata.36561 2024-03-25T23:10:29.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-25T23:10:29.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj20 testdata.36561 2024-03-25T23:10:30.816 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-25T23:10:30.816 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj21 testdata.36561 2024-03-25T23:10:32.614 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-25T23:10:32.614 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj22 testdata.36561 2024-03-25T23:10:34.455 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-25T23:10:34.455 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj23 testdata.36561 2024-03-25T23:10:36.403 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-25T23:10:36.403 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj24 testdata.36561 2024-03-25T23:10:38.219 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-25T23:10:38.219 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj25 testdata.36561 2024-03-25T23:10:40.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-25T23:10:40.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj26 testdata.36561 2024-03-25T23:10:42.008 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-25T23:10:42.008 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj27 testdata.36561 2024-03-25T23:10:43.875 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-25T23:10:43.875 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj28 testdata.36561 2024-03-25T23:10:45.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-25T23:10:45.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj29 testdata.36561 2024-03-25T23:10:47.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-25T23:10:47.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj30 testdata.36561 2024-03-25T23:10:49.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-25T23:10:49.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj31 testdata.36561 2024-03-25T23:10:51.277 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-25T23:10:51.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj32 testdata.36561 2024-03-25T23:10:53.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-25T23:10:53.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj33 testdata.36561 2024-03-25T23:10:55.002 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-25T23:10:55.002 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj34 testdata.36561 2024-03-25T23:10:56.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-25T23:10:56.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj35 testdata.36561 2024-03-25T23:10:58.683 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-25T23:10:58.683 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj36 testdata.36561 2024-03-25T23:11:00.524 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-25T23:11:00.524 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj37 testdata.36561 2024-03-25T23:11:02.363 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-25T23:11:02.364 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj38 testdata.36561 2024-03-25T23:11:04.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-25T23:11:04.247 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj39 testdata.36561 2024-03-25T23:11:06.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-25T23:11:06.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj40 testdata.36561 2024-03-25T23:11:08.127 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:265: TEST_recovery_scrub_2: rm -f testdata.36561 2024-03-25T23:11:08.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:267: TEST_recovery_scrub_2: ceph osd pool set test size 3 2024-03-25T23:11:08.549 INFO:tasks.workunit.client.0.smithi129.stderr:set pool 1 size to 3 2024-03-25T23:11:08.570 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:269: TEST_recovery_scrub_2: ceph pg dump pgs 2024-03-25T23:11:08.879 INFO:tasks.workunit.client.0.smithi129.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-25T23:11:08.879 INFO:tasks.workunit.client.0.smithi129.stdout:1.1f 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-03-25T23:09:46.304502+0000 47'26 47:61 [5] 5 [5] 5 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:09:44.508877+0000 0 0 2024-03-25T23:11:08.879 INFO:tasks.workunit.client.0.smithi129.stdout:1.1e 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-25T23:09:47.356834+0000 47'13 47:46 [3] 3 [3] 3 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:09:44.508877+0000 0 0 2024-03-25T23:11:08.879 INFO:tasks.workunit.client.0.smithi129.stdout:1.1d 2 0 0 0 0 65011712 0 0 16 0 16 active+clean 2024-03-25T23:09:46.180887+0000 47'16 47:36 [1] 1 [1] 1 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:09:44.508877+0000 0 0 2024-03-25T23:11:08.879 INFO:tasks.workunit.client.0.smithi129.stdout:1.b 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-25T23:09:46.179717+0000 47'13 47:36 [1] 1 [1] 1 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:09:44.508877+0000 0 0 2024-03-25T23:11:08.879 INFO:tasks.workunit.client.0.smithi129.stdout:1.a 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-25T23:09:46.304782+0000 47'13 47:36 [1] 1 [1] 1 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:09:44.508877+0000 0 0 2024-03-25T23:11:08.879 INFO:tasks.workunit.client.0.smithi129.stdout:1.9 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T23:09:46.179625+0000 0'0 47:23 [1] 1 [1] 1 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:09:44.508877+0000 0 0 2024-03-25T23:11:08.880 INFO:tasks.workunit.client.0.smithi129.stdout:1.8 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-03-25T23:09:46.179025+0000 47'26 47:50 [5] 5 [5] 5 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:09:44.508877+0000 0 0 2024-03-25T23:11:08.880 INFO:tasks.workunit.client.0.smithi129.stdout:1.7 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-25T23:09:46.763297+0000 47'13 47:36 [6] 6 [6] 6 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:09:44.508877+0000 0 0 2024-03-25T23:11:08.880 INFO:tasks.workunit.client.0.smithi129.stdout:1.6 1 0 0 0 0 50331648 0 0 12 0 12 active+clean 2024-03-25T23:09:46.762711+0000 47'12 47:36 [6] 6 [6] 6 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:09:44.508877+0000 0 0 2024-03-25T23:11:08.880 INFO:tasks.workunit.client.0.smithi129.stdout:1.5 3 0 0 0 0 157286400 0 0 39 0 39 active+clean 2024-03-25T23:09:46.895720+0000 47'39 47:63 [4] 4 [4] 4 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:09:44.508877+0000 0 0 2024-03-25T23:11:08.880 INFO:tasks.workunit.client.0.smithi129.stdout:1.4 4 0 0 0 0 209715200 0 0 52 0 52 active+clean 2024-03-25T23:09:47.243699+0000 47'52 47:77 [3] 3 [3] 3 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:09:44.508877+0000 0 0 2024-03-25T23:11:08.880 INFO:tasks.workunit.client.0.smithi129.stdout:1.3 4 0 0 0 0 209715200 0 0 52 0 52 active+clean 2024-03-25T23:09:46.762713+0000 47'52 47:77 [4] 4 [4] 4 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:09:44.508877+0000 0 0 2024-03-25T23:11:08.880 INFO:tasks.workunit.client.0.smithi129.stdout:1.2 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-25T23:09:46.145040+0000 47'13 47:39 [0] 0 [0] 0 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:09:44.508877+0000 0 0 2024-03-25T23:11:08.880 INFO:tasks.workunit.client.0.smithi129.stdout:1.c 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T23:09:47.356524+0000 0'0 47:21 [3] 3 [3] 3 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:09:44.508877+0000 0 0 2024-03-25T23:11:08.880 INFO:tasks.workunit.client.0.smithi129.stdout:1.1 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-25T23:09:46.179074+0000 47'13 47:48 [5] 5 [5] 5 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:09:44.508877+0000 0 0 2024-03-25T23:11:08.880 INFO:tasks.workunit.client.0.smithi129.stdout:1.1c 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-25T23:09:46.306409+0000 47'13 47:32 [0] 0 [0] 0 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:09:44.508877+0000 0 0 2024-03-25T23:11:08.880 INFO:tasks.workunit.client.0.smithi129.stdout:1.0 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-25T23:09:45.599964+0000 47'13 47:47 [7] 7 [7] 7 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:09:44.508877+0000 0 0 2024-03-25T23:11:08.881 INFO:tasks.workunit.client.0.smithi129.stdout:1.d 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-25T23:09:45.759190+0000 47'13 47:36 [7] 7 [7] 7 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:09:44.508877+0000 0 0 2024-03-25T23:11:08.881 INFO:tasks.workunit.client.0.smithi129.stdout:1.e 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-25T23:09:46.180621+0000 47'13 47:35 [1] 1 [1] 1 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:09:44.508877+0000 0 0 2024-03-25T23:11:08.881 INFO:tasks.workunit.client.0.smithi129.stdout:1.f 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-25T23:09:46.762908+0000 47'13 47:34 [6] 6 [6] 6 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:09:44.508877+0000 0 0 2024-03-25T23:11:08.881 INFO:tasks.workunit.client.0.smithi129.stdout:1.10 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-25T23:09:46.304324+0000 47'13 47:36 [5] 5 [5] 5 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:09:44.508877+0000 0 0 2024-03-25T23:11:08.881 INFO:tasks.workunit.client.0.smithi129.stdout:1.11 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-25T23:09:46.304972+0000 47'13 47:33 [0] 0 [0] 0 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:09:44.508877+0000 0 0 2024-03-25T23:11:08.881 INFO:tasks.workunit.client.0.smithi129.stdout:1.12 2 0 0 0 0 102760448 0 0 25 0 25 active+clean 2024-03-25T23:09:46.305811+0000 47'25 47:45 [0] 0 [0] 0 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:09:44.508877+0000 0 0 2024-03-25T23:11:08.881 INFO:tasks.workunit.client.0.smithi129.stdout:1.13 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-25T23:09:46.762815+0000 47'13 47:33 [4] 4 [4] 4 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:09:44.508877+0000 0 0 2024-03-25T23:11:08.881 INFO:tasks.workunit.client.0.smithi129.stdout:1.14 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T23:09:46.145118+0000 0'0 47:21 [2] 2 [2] 2 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:09:44.508877+0000 0 0 2024-03-25T23:11:08.881 INFO:tasks.workunit.client.0.smithi129.stdout:1.15 3 0 0 0 0 157286400 0 0 39 0 39 active+clean 2024-03-25T23:09:46.180182+0000 47'39 47:60 [1] 1 [1] 1 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:09:44.508877+0000 0 0 2024-03-25T23:11:08.881 INFO:tasks.workunit.client.0.smithi129.stdout:1.16 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T23:09:46.763542+0000 0'0 47:20 [6] 6 [6] 6 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:09:44.508877+0000 0 0 2024-03-25T23:11:08.881 INFO:tasks.workunit.client.0.smithi129.stdout:1.17 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-25T23:09:46.306103+0000 47'13 47:32 [0] 0 [0] 0 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:09:44.508877+0000 0 0 2024-03-25T23:11:08.881 INFO:tasks.workunit.client.0.smithi129.stdout:1.18 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T23:09:46.180288+0000 0'0 47:21 [5] 5 [5] 5 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:09:44.508877+0000 0 0 2024-03-25T23:11:08.881 INFO:tasks.workunit.client.0.smithi129.stdout:1.19 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-25T23:09:47.356717+0000 47'13 47:32 [3] 3 [3] 3 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:09:44.508877+0000 0 0 2024-03-25T23:11:08.882 INFO:tasks.workunit.client.0.smithi129.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T23:09:46.304192+0000 0'0 47:20 [1] 1 [1] 1 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:09:44.508877+0000 0 0 2024-03-25T23:11:08.882 INFO:tasks.workunit.client.0.smithi129.stdout:1.1b 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-25T23:09:46.305111+0000 47'13 47:32 [0] 0 [0] 0 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:09:44.508877+0000 0 0 2024-03-25T23:11:08.882 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:11:08.882 INFO:tasks.workunit.client.0.smithi129.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-25T23:11:08.882 INFO:tasks.workunit.client.0.smithi129.stderr:dumped pgs 2024-03-25T23:11:08.895 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:275: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active 2024-03-25T23:11:09.033 INFO:tasks.workunit.client.0.smithi129.stdout:{ 2024-03-25T23:11:09.033 INFO:tasks.workunit.client.0.smithi129.stdout: "osd_recovery_max_active": "0" 2024-03-25T23:11:09.033 INFO:tasks.workunit.client.0.smithi129.stdout:} 2024-03-25T23:11:09.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:277: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active_hdd 2024-03-25T23:11:09.366 INFO:tasks.workunit.client.0.smithi129.stdout:{ 2024-03-25T23:11:09.366 INFO:tasks.workunit.client.0.smithi129.stdout: "osd_recovery_max_active_hdd": "3" 2024-03-25T23:11:09.366 INFO:tasks.workunit.client.0.smithi129.stdout:} 2024-03-25T23:11:09.384 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:278: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active_ssd 2024-03-25T23:11:09.685 INFO:tasks.workunit.client.0.smithi129.stdout:{ 2024-03-25T23:11:09.686 INFO:tasks.workunit.client.0.smithi129.stdout: "osd_recovery_max_active_ssd": "10" 2024-03-25T23:11:09.686 INFO:tasks.workunit.client.0.smithi129.stdout:} 2024-03-25T23:11:09.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:281: TEST_recovery_scrub_2: count=0 2024-03-25T23:11:09.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-03-25T23:11:09.705 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-03-25T23:11:09.705 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-03-25T23:11:09.706 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-03-25T23:11:09.706 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-03-25T23:11:10.292 INFO:tasks.workunit.client.0.smithi129.stderr:dumped pgs 2024-03-25T23:11:10.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 0 -ge 2 2024-03-25T23:11:10.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-03-25T23:11:12.311 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 0 -eq 10 2024-03-25T23:11:12.311 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 0 + 1 2024-03-25T23:11:12.312 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=1 2024-03-25T23:11:12.313 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-03-25T23:11:12.314 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-03-25T23:11:12.314 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-03-25T23:11:12.314 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-03-25T23:11:12.314 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-03-25T23:11:12.615 INFO:tasks.workunit.client.0.smithi129.stderr:dumped pgs 2024-03-25T23:11:12.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 0 -ge 2 2024-03-25T23:11:12.635 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-03-25T23:11:14.636 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 1 -eq 10 2024-03-25T23:11:14.637 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 1 + 1 2024-03-25T23:11:14.638 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=2 2024-03-25T23:11:14.639 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-03-25T23:11:14.640 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-03-25T23:11:14.641 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-03-25T23:11:14.641 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-03-25T23:11:14.641 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-03-25T23:11:14.942 INFO:tasks.workunit.client.0.smithi129.stderr:dumped pgs 2024-03-25T23:11:14.959 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 2 -ge 2 2024-03-25T23:11:14.959 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:288: TEST_recovery_scrub_2: break 2024-03-25T23:11:14.959 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:298: TEST_recovery_scrub_2: ceph pg dump pgs 2024-03-25T23:11:15.239 INFO:tasks.workunit.client.0.smithi129.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-25T23:11:15.239 INFO:tasks.workunit.client.0.smithi129.stdout:1.1f 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-03-25T23:11:11.762482+0000 47'26 50:71 [5,0,2] 5 [5,0,2] 5 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 no scrub is scheduled 0 0 2024-03-25T23:11:15.239 INFO:tasks.workunit.client.0.smithi129.stdout:1.1e 1 0 2 0 0 52428800 0 0 13 0 13 active+recovering+degraded 2024-03-25T23:11:11.555186+0000 47'13 50:57 [3,0,7] 3 [3,0,7] 3 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 no scrub is scheduled 0 0 2024-03-25T23:11:15.239 INFO:tasks.workunit.client.0.smithi129.stdout:1.1d 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-03-25T23:11:11.334020+0000 47'26 50:55 [1,3,0] 1 [1,3,0] 1 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 no scrub is scheduled 0 0 2024-03-25T23:11:15.239 INFO:tasks.workunit.client.0.smithi129.stdout:1.b 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-25T23:11:11.552357+0000 47'13 50:45 [1,0,2] 1 [1,0,2] 1 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 no scrub is scheduled 0 0 2024-03-25T23:11:15.239 INFO:tasks.workunit.client.0.smithi129.stdout:1.a 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-25T23:11:11.551310+0000 47'13 50:45 [1,6,2] 1 [1,6,2] 1 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 no scrub is scheduled 0 0 2024-03-25T23:11:15.240 INFO:tasks.workunit.client.0.smithi129.stdout:1.9 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T23:11:11.296978+0000 0'0 50:34 [1,4,5] 1 [1,4,5] 1 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:09:44.508877+0000 0 0 2024-03-25T23:11:15.240 INFO:tasks.workunit.client.0.smithi129.stdout:1.8 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-03-25T23:11:11.296841+0000 47'26 50:58 [5,4,3] 5 [5,4,3] 5 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 no scrub is scheduled 0 0 2024-03-25T23:11:15.240 INFO:tasks.workunit.client.0.smithi129.stdout:1.7 1 0 2 0 0 52428800 0 0 13 0 13 activating 2024-03-25T23:11:10.409900+0000 47'13 50:40 [6,7,3] 6 [6,7,3] 6 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 no scrub is scheduled 0 0 2024-03-25T23:11:15.240 INFO:tasks.workunit.client.0.smithi129.stdout:1.6 1 0 2 0 0 52428800 0 0 13 0 13 activating 2024-03-25T23:11:10.408733+0000 47'13 50:42 [6,7,3] 6 [6,7,3] 6 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 no scrub is scheduled 0 0 2024-03-25T23:11:15.240 INFO:tasks.workunit.client.0.smithi129.stdout:1.5 3 0 6 0 0 157286400 0 0 39 0 39 activating+degraded 2024-03-25T23:11:10.116820+0000 47'39 49:67 [4,2,6] 4 [4,2,6] 4 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 no scrub is scheduled 0 0 2024-03-25T23:11:15.240 INFO:tasks.workunit.client.0.smithi129.stdout:1.4 4 0 8 0 0 209715200 0 0 52 0 52 active+recovery_wait+degraded 2024-03-25T23:11:11.553933+0000 47'52 50:86 [3,0,6] 3 [3,0,6] 3 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 no scrub is scheduled 0 0 2024-03-25T23:11:15.240 INFO:tasks.workunit.client.0.smithi129.stdout:1.3 4 0 8 0 0 209715200 0 0 52 0 52 activating 2024-03-25T23:11:09.555720+0000 47'52 49:80 [4,1,2] 4 [4,1,2] 4 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 no scrub is scheduled 0 0 2024-03-25T23:11:15.240 INFO:tasks.workunit.client.0.smithi129.stdout:1.2 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-25T23:11:11.142375+0000 47'13 50:50 [0,7,1] 0 [0,7,1] 0 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 no scrub is scheduled 0 0 2024-03-25T23:11:15.240 INFO:tasks.workunit.client.0.smithi129.stdout:1.c 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T23:11:11.334325+0000 0'0 50:33 [3,0,4] 3 [3,0,4] 3 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:09:44.508877+0000 0 0 2024-03-25T23:11:15.240 INFO:tasks.workunit.client.0.smithi129.stdout:1.1 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-25T23:11:11.762754+0000 47'13 50:58 [5,0,4] 5 [5,0,4] 5 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 no scrub is scheduled 0 0 2024-03-25T23:11:15.240 INFO:tasks.workunit.client.0.smithi129.stdout:1.1c 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-25T23:11:10.460945+0000 47'13 50:42 [0,3,4] 0 [0,3,4] 0 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 no scrub is scheduled 0 0 2024-03-25T23:11:15.240 INFO:tasks.workunit.client.0.smithi129.stdout:1.0 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-25T23:11:10.058082+0000 47'13 50:59 [7,0,2] 7 [7,0,2] 7 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 no scrub is scheduled 0 0 2024-03-25T23:11:15.241 INFO:tasks.workunit.client.0.smithi129.stdout:1.d 1 0 2 0 0 52428800 0 0 13 0 13 active+recovering+degraded 2024-03-25T23:11:10.059979+0000 47'13 50:47 [7,4,2] 7 [7,4,2] 7 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 no scrub is scheduled 0 0 2024-03-25T23:11:15.241 INFO:tasks.workunit.client.0.smithi129.stdout:1.e 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-25T23:11:11.551892+0000 47'13 50:44 [1,4,2] 1 [1,4,2] 1 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 no scrub is scheduled 0 0 2024-03-25T23:11:15.241 INFO:tasks.workunit.client.0.smithi129.stdout:1.f 1 0 2 0 0 52428800 0 0 13 0 13 activating 2024-03-25T23:11:10.408902+0000 47'13 50:37 [6,0,3] 6 [6,0,3] 6 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 no scrub is scheduled 0 0 2024-03-25T23:11:15.241 INFO:tasks.workunit.client.0.smithi129.stdout:1.10 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-25T23:11:11.555466+0000 47'13 50:45 [5,7,6] 5 [5,7,6] 5 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 no scrub is scheduled 0 0 2024-03-25T23:11:15.241 INFO:tasks.workunit.client.0.smithi129.stdout:1.11 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-25T23:11:10.934182+0000 47'13 50:43 [0,1,3] 0 [0,1,3] 0 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 no scrub is scheduled 0 0 2024-03-25T23:11:15.241 INFO:tasks.workunit.client.0.smithi129.stdout:1.12 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-03-25T23:11:10.934803+0000 47'26 50:56 [0,1,6] 0 [0,1,6] 0 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 no scrub is scheduled 0 0 2024-03-25T23:11:15.241 INFO:tasks.workunit.client.0.smithi129.stdout:1.13 1 0 2 0 0 52428800 0 0 13 0 13 activating 2024-03-25T23:11:09.554383+0000 47'13 49:36 [4,6,3] 4 [4,6,3] 4 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 no scrub is scheduled 0 0 2024-03-25T23:11:15.241 INFO:tasks.workunit.client.0.smithi129.stdout:1.14 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T23:11:10.057626+0000 0'0 50:33 [2,0,4] 2 [2,0,4] 2 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:09:44.508877+0000 0 0 2024-03-25T23:11:15.241 INFO:tasks.workunit.client.0.smithi129.stdout:1.15 3 0 6 0 0 157286400 0 0 39 0 39 active+recovery_wait+degraded 2024-03-25T23:11:11.144407+0000 47'39 50:69 [1,5,3] 1 [1,5,3] 1 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 no scrub is scheduled 0 0 2024-03-25T23:11:15.241 INFO:tasks.workunit.client.0.smithi129.stdout:1.16 0 0 0 0 0 0 0 0 0 0 0 activating 2024-03-25T23:11:10.408452+0000 0'0 50:25 [6,0,7] 6 [6,0,7] 6 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 no scrub is scheduled 0 0 2024-03-25T23:11:15.241 INFO:tasks.workunit.client.0.smithi129.stdout:1.17 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-25T23:11:10.935165+0000 47'13 50:43 [0,4,5] 0 [0,4,5] 0 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 no scrub is scheduled 0 0 2024-03-25T23:11:15.241 INFO:tasks.workunit.client.0.smithi129.stdout:1.18 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T23:11:11.551154+0000 0'0 50:32 [5,2,3] 5 [5,2,3] 5 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:09:44.508877+0000 0 0 2024-03-25T23:11:15.241 INFO:tasks.workunit.client.0.smithi129.stdout:1.19 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-25T23:11:11.761242+0000 47'13 50:41 [3,0,4] 3 [3,0,4] 3 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 no scrub is scheduled 0 0 2024-03-25T23:11:15.241 INFO:tasks.workunit.client.0.smithi129.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T23:11:11.144877+0000 0'0 50:31 [1,7,4] 1 [1,7,4] 1 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:09:44.508877+0000 0 0 2024-03-25T23:11:15.242 INFO:tasks.workunit.client.0.smithi129.stdout:1.1b 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-25T23:11:10.933774+0000 47'13 50:42 [0,1,2] 0 [0,1,2] 0 0'0 2024-03-25T23:09:44.508877+0000 0'0 2024-03-25T23:09:44.508877+0000 0 0 no scrub is scheduled 0 0 2024-03-25T23:11:15.242 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:11:15.242 INFO:tasks.workunit.client.0.smithi129.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-25T23:11:15.242 INFO:tasks.workunit.client.0.smithi129.stderr:dumped pgs 2024-03-25T23:11:15.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:300: TEST_recovery_scrub_2: pids= 2024-03-25T23:11:15.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:301: TEST_recovery_scrub_2: recov_scrub_count=0 2024-03-25T23:11:15.255 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: expr 32 - 1 2024-03-25T23:11:15.257 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: seq 0 31 2024-03-25T23:11:15.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-25T23:11:15.258 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 0 2024-03-25T23:11:15.259 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.0 2024-03-25T23:11:15.259 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:11:15.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:11:15.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58182"' 2024-03-25T23:11:15.260 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58182' 2024-03-25T23:11:15.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-25T23:11:15.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.0 2024-03-25T23:11:15.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58185: /' 2024-03-25T23:11:15.260 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 1 2024-03-25T23:11:15.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1 2024-03-25T23:11:15.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:11:15.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:11:15.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58187"' 2024-03-25T23:11:15.262 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58187' 2024-03-25T23:11:15.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-25T23:11:15.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1 2024-03-25T23:11:15.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58192: /' 2024-03-25T23:11:15.262 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 2 2024-03-25T23:11:15.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.2 2024-03-25T23:11:15.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:11:15.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:11:15.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58195"' 2024-03-25T23:11:15.263 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58195' 2024-03-25T23:11:15.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-25T23:11:15.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.2 2024-03-25T23:11:15.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58199: /' 2024-03-25T23:11:15.264 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 3 2024-03-25T23:11:15.264 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.3 2024-03-25T23:11:15.264 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:11:15.264 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:11:15.264 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58201"' 2024-03-25T23:11:15.264 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58201' 2024-03-25T23:11:15.264 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-25T23:11:15.264 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 4 2024-03-25T23:11:15.264 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58206: /' 2024-03-25T23:11:15.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.4 2024-03-25T23:11:15.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:11:15.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:11:15.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58207"' 2024-03-25T23:11:15.265 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58207' 2024-03-25T23:11:15.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-25T23:11:15.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.3 2024-03-25T23:11:15.271 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.4 2024-03-25T23:11:15.277 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 5 2024-03-25T23:11:15.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.5 2024-03-25T23:11:15.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:11:15.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:11:15.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58215"' 2024-03-25T23:11:15.279 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58215' 2024-03-25T23:11:15.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-25T23:11:15.285 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58213: /' 2024-03-25T23:11:15.285 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 6 2024-03-25T23:11:15.286 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.6 2024-03-25T23:11:15.286 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:11:15.286 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:11:15.286 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58217"' 2024-03-25T23:11:15.286 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58217' 2024-03-25T23:11:15.287 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-25T23:11:15.293 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 7 2024-03-25T23:11:15.294 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.7 2024-03-25T23:11:15.294 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:11:15.294 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:11:15.294 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58223"' 2024-03-25T23:11:15.294 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58223' 2024-03-25T23:11:15.294 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-25T23:11:15.296 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.5 2024-03-25T23:11:15.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58220: /' 2024-03-25T23:11:15.301 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 8 2024-03-25T23:11:15.302 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.8 2024-03-25T23:11:15.302 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:11:15.302 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:11:15.302 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58230"' 2024-03-25T23:11:15.302 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58230' 2024-03-25T23:11:15.302 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-25T23:11:15.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58227: /' 2024-03-25T23:11:15.303 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 9 2024-03-25T23:11:15.304 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.9 2024-03-25T23:11:15.304 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:11:15.304 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:11:15.304 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58234"' 2024-03-25T23:11:15.304 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58234' 2024-03-25T23:11:15.304 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-25T23:11:15.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58229: /' 2024-03-25T23:11:15.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.7 2024-03-25T23:11:15.313 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.6 2024-03-25T23:11:15.313 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 10 2024-03-25T23:11:15.313 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.a 2024-03-25T23:11:15.314 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:11:15.314 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:11:15.314 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58241"' 2024-03-25T23:11:15.314 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58241' 2024-03-25T23:11:15.314 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-25T23:11:15.320 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58245: /' 2024-03-25T23:11:15.321 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.9 2024-03-25T23:11:15.321 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 11 2024-03-25T23:11:15.321 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58237: /' 2024-03-25T23:11:15.321 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.b 2024-03-25T23:11:15.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:11:15.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:11:15.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58247"' 2024-03-25T23:11:15.326 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58247' 2024-03-25T23:11:15.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-25T23:11:15.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.8 2024-03-25T23:11:15.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.b 2024-03-25T23:11:15.338 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58255: /' 2024-03-25T23:11:15.341 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 12 2024-03-25T23:11:15.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.a 2024-03-25T23:11:15.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.c 2024-03-25T23:11:15.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:11:15.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:11:15.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58262"' 2024-03-25T23:11:15.350 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58262' 2024-03-25T23:11:15.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-25T23:11:15.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58251: /' 2024-03-25T23:11:15.358 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58265: /' 2024-03-25T23:11:15.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.c 2024-03-25T23:11:15.369 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 13 2024-03-25T23:11:15.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.d 2024-03-25T23:11:15.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:11:15.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:11:15.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58271"' 2024-03-25T23:11:15.370 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58271' 2024-03-25T23:11:15.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-25T23:11:15.377 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 14 2024-03-25T23:11:15.378 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.e 2024-03-25T23:11:15.378 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:11:15.378 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:11:15.378 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58275"' 2024-03-25T23:11:15.378 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58275' 2024-03-25T23:11:15.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-25T23:11:15.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58278: /' 2024-03-25T23:11:15.393 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 15 2024-03-25T23:11:15.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.f 2024-03-25T23:11:15.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:11:15.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:11:15.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58281"' 2024-03-25T23:11:15.394 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58281' 2024-03-25T23:11:15.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-25T23:11:15.400 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.d 2024-03-25T23:11:15.400 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58280: /' 2024-03-25T23:11:15.402 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.e 2024-03-25T23:11:15.417 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58284: /' 2024-03-25T23:11:15.417 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 16 2024-03-25T23:11:15.417 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.10 2024-03-25T23:11:15.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:11:15.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:11:15.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58291"' 2024-03-25T23:11:15.418 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58291' 2024-03-25T23:11:15.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-25T23:11:15.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.f 2024-03-25T23:11:15.419 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 17 2024-03-25T23:11:15.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.11 2024-03-25T23:11:15.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:11:15.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:11:15.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58294"' 2024-03-25T23:11:15.420 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58294' 2024-03-25T23:11:15.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-25T23:11:15.429 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58297: /' 2024-03-25T23:11:15.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.10 2024-03-25T23:11:15.438 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 18 2024-03-25T23:11:15.438 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.11 2024-03-25T23:11:15.438 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.12 2024-03-25T23:11:15.439 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:11:15.439 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:11:15.439 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58301"' 2024-03-25T23:11:15.439 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58301' 2024-03-25T23:11:15.439 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-25T23:11:15.445 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58300: /' 2024-03-25T23:11:15.457 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 19 2024-03-25T23:11:15.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.13 2024-03-25T23:11:15.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:11:15.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:11:15.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58310"' 2024-03-25T23:11:15.458 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58310' 2024-03-25T23:11:15.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-25T23:11:15.473 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 20 2024-03-25T23:11:15.474 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.14 2024-03-25T23:11:15.474 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:11:15.474 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:11:15.474 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58313"' 2024-03-25T23:11:15.474 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58313' 2024-03-25T23:11:15.474 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-25T23:11:15.481 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 21 2024-03-25T23:11:15.482 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.15 2024-03-25T23:11:15.482 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:11:15.482 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:11:15.482 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58319"' 2024-03-25T23:11:15.482 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58319' 2024-03-25T23:11:15.482 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-25T23:11:15.482 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58316: /' 2024-03-25T23:11:15.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.12 2024-03-25T23:11:15.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.13 2024-03-25T23:11:15.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58318: /' 2024-03-25T23:11:15.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.14 2024-03-25T23:11:15.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58326: /' 2024-03-25T23:11:15.501 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.15 2024-03-25T23:11:15.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58323: /' 2024-03-25T23:11:15.512 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 22 2024-03-25T23:11:15.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.16 2024-03-25T23:11:15.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:11:15.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:11:15.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58337"' 2024-03-25T23:11:15.513 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58337' 2024-03-25T23:11:15.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-25T23:11:15.521 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 23 2024-03-25T23:11:15.529 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.17 2024-03-25T23:11:15.529 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:11:15.529 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:11:15.530 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58341"' 2024-03-25T23:11:15.530 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58341' 2024-03-25T23:11:15.530 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-25T23:11:15.549 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58344: /' 2024-03-25T23:11:15.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.16 2024-03-25T23:11:15.553 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 24 2024-03-25T23:11:15.554 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.18 2024-03-25T23:11:15.554 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:11:15.554 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:11:15.554 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58348"' 2024-03-25T23:11:15.554 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58348' 2024-03-25T23:11:15.554 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-25T23:11:15.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.17 2024-03-25T23:11:15.570 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 25 2024-03-25T23:11:15.570 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.19 2024-03-25T23:11:15.570 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:11:15.571 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:11:15.571 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58357"' 2024-03-25T23:11:15.571 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58357' 2024-03-25T23:11:15.571 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-25T23:11:15.571 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58356: /' 2024-03-25T23:11:15.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.18 2024-03-25T23:11:15.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58346: /' 2024-03-25T23:11:15.589 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58362: /' 2024-03-25T23:11:15.597 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 26 2024-03-25T23:11:15.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1a 2024-03-25T23:11:15.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:11:15.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:11:15.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58369"' 2024-03-25T23:11:15.598 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58369' 2024-03-25T23:11:15.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-25T23:11:15.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.19 2024-03-25T23:11:15.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58374: /' 2024-03-25T23:11:15.614 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1a 2024-03-25T23:11:15.615 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 27 2024-03-25T23:11:15.621 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1b 2024-03-25T23:11:15.621 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:11:15.621 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:11:15.621 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58380"' 2024-03-25T23:11:15.621 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58380' 2024-03-25T23:11:15.621 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-25T23:11:15.640 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 28 2024-03-25T23:11:15.642 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1c 2024-03-25T23:11:15.642 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:11:15.642 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:11:15.642 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58393"' 2024-03-25T23:11:15.642 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58393' 2024-03-25T23:11:15.642 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-25T23:11:15.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1b 2024-03-25T23:11:15.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58386: /' 2024-03-25T23:11:15.657 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 29 2024-03-25T23:11:15.658 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1d 2024-03-25T23:11:15.658 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:11:15.658 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:11:15.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58412"' 2024-03-25T23:11:15.659 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58412' 2024-03-25T23:11:15.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-25T23:11:15.668 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58409: /' 2024-03-25T23:11:15.672 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1c 2024-03-25T23:11:15.673 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 30 2024-03-25T23:11:15.674 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1e 2024-03-25T23:11:15.674 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:11:15.675 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:11:15.675 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58417"' 2024-03-25T23:11:15.675 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58417' 2024-03-25T23:11:15.675 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-25T23:11:15.687 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58420: /' 2024-03-25T23:11:15.693 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1d 2024-03-25T23:11:15.693 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 31 2024-03-25T23:11:15.694 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1f 2024-03-25T23:11:15.694 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:11:15.694 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:11:15.694 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58432"' 2024-03-25T23:11:15.694 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58432' 2024-03-25T23:11:15.694 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:306: TEST_recovery_scrub_2: wait_background_check pids 2024-03-25T23:11:15.695 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: pids=' 58182 58187 58195 58201 58207 58215 58217 58223 58230 58234 58241 58247 58262 58271 58275 58281 58291 58294 58301 58310 58313 58319 58337 58341 58348 58357 58369 58380 58393 58412 58417 58432' 2024-03-25T23:11:15.695 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: return_code=0 2024-03-25T23:11:15.695 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-25T23:11:15.695 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58182 2024-03-25T23:11:15.717 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58430: /' 2024-03-25T23:11:15.717 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1e 2024-03-25T23:11:15.745 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58440: /' 2024-03-25T23:11:15.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1f 2024-03-25T23:11:22.749 INFO:tasks.workunit.client.0.smithi129.stderr:58280: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.e 2024-03-25T23:11:22.749 INFO:tasks.workunit.client.0.smithi129.stderr:58280: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.e 2024-03-25T23:11:22.750 INFO:tasks.workunit.client.0.smithi129.stderr:58280: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.e 2024-03-25T23:11:22.750 INFO:tasks.workunit.client.0.smithi129.stderr:58280: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:22.750 INFO:tasks.workunit.client.0.smithi129.stderr:58280: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2024-03-25T23:11:22.750 INFO:tasks.workunit.client.0.smithi129.stderr:58280: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:22.750 INFO:tasks.workunit.client.0.smithi129.stderr:58280: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:22.750 INFO:tasks.workunit.client.0.smithi129.stderr:58280: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-25T23:11:22.750 INFO:tasks.workunit.client.0.smithi129.stderr:58280: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.e query 2024-03-25T23:11:22.750 INFO:tasks.workunit.client.0.smithi129.stderr:58280: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-25T23:11:22.750 INFO:tasks.workunit.client.0.smithi129.stderr:58280: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-25T23:11:22.750 INFO:tasks.workunit.client.0.smithi129.stderr:58280: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-25T23:11:22.750 INFO:tasks.workunit.client.0.smithi129.stderr:58280: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.e 2024-03-25T23:11:22.750 INFO:tasks.workunit.client.0.smithi129.stderr:58280: instructing pg 1.e on osd.1 to scrub 2024-03-25T23:11:22.751 INFO:tasks.workunit.client.0.smithi129.stderr:58280: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-25T23:11:22.751 INFO:tasks.workunit.client.0.smithi129.stderr:58280: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-25T23:11:22.751 INFO:tasks.workunit.client.0.smithi129.stderr:58280: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.e") | .state' 2024-03-25T23:11:22.751 INFO:tasks.workunit.client.0.smithi129.stderr:58280: dumped pgs 2024-03-25T23:11:22.751 INFO:tasks.workunit.client.0.smithi129.stderr:58280: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.e 2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:22.751 INFO:tasks.workunit.client.0.smithi129.stderr:58280: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.e 2024-03-25T23:11:22.751 INFO:tasks.workunit.client.0.smithi129.stderr:58280: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:22.751 INFO:tasks.workunit.client.0.smithi129.stderr:58280: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-25T23:11:22.751 INFO:tasks.workunit.client.0.smithi129.stderr:58280: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-25T23:11:22.751 INFO:tasks.workunit.client.0.smithi129.stderr:58280: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-25T23:11:22.751 INFO:tasks.workunit.client.0.smithi129.stderr:58280: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-25T23:11:22.751 INFO:tasks.workunit.client.0.smithi129.stderr:58280: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-25T23:11:22.751 INFO:tasks.workunit.client.0.smithi129.stderr:58280: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.e last_scrub_stamp 2024-03-25T23:11:22.751 INFO:tasks.workunit.client.0.smithi129.stderr:58280: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.e 2024-03-25T23:11:22.752 INFO:tasks.workunit.client.0.smithi129.stderr:58280: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:22.752 INFO:tasks.workunit.client.0.smithi129.stderr:58280: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:22.752 INFO:tasks.workunit.client.0.smithi129.stderr:58280: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2024-03-25T23:11:22.752 INFO:tasks.workunit.client.0.smithi129.stderr:58280: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-25T23:09:44.508877+0000 '>' '' 2024-03-25T23:11:22.752 INFO:tasks.workunit.client.0.smithi129.stderr:58280: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-25T23:11:22.752 INFO:tasks.workunit.client.0.smithi129.stderr:58280: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-25T23:11:22.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:11:23.309 INFO:tasks.workunit.client.0.smithi129.stderr:58430: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1e 2024-03-25T23:11:23.309 INFO:tasks.workunit.client.0.smithi129.stderr:58430: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1e 2024-03-25T23:11:23.309 INFO:tasks.workunit.client.0.smithi129.stderr:58430: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1e 2024-03-25T23:11:23.309 INFO:tasks.workunit.client.0.smithi129.stderr:58430: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:23.310 INFO:tasks.workunit.client.0.smithi129.stderr:58430: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:23.310 INFO:tasks.workunit.client.0.smithi129.stderr:58430: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2024-03-25T23:11:23.310 INFO:tasks.workunit.client.0.smithi129.stderr:58430: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:23.310 INFO:tasks.workunit.client.0.smithi129.stderr:58430: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-25T23:11:23.310 INFO:tasks.workunit.client.0.smithi129.stderr:58430: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1e query 2024-03-25T23:11:23.310 INFO:tasks.workunit.client.0.smithi129.stderr:58430: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-25T23:11:23.310 INFO:tasks.workunit.client.0.smithi129.stderr:58430: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-25T23:11:23.310 INFO:tasks.workunit.client.0.smithi129.stderr:58430: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-25T23:11:23.310 INFO:tasks.workunit.client.0.smithi129.stderr:58430: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1e 2024-03-25T23:11:23.310 INFO:tasks.workunit.client.0.smithi129.stderr:58430: instructing pg 1.1e on osd.3 to scrub 2024-03-25T23:11:23.314 INFO:tasks.workunit.client.0.smithi129.stderr:58430: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-25T23:11:23.314 INFO:tasks.workunit.client.0.smithi129.stderr:58430: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-25T23:11:23.314 INFO:tasks.workunit.client.0.smithi129.stderr:58430: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1e") | .state' 2024-03-25T23:11:23.314 INFO:tasks.workunit.client.0.smithi129.stderr:58430: dumped pgs 2024-03-25T23:11:23.314 INFO:tasks.workunit.client.0.smithi129.stderr:58430: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:198: pg_scrub_mod: recovery=true 2024-03-25T23:11:23.314 INFO:tasks.workunit.client.0.smithi129.stderr:58430: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1e 2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:23.314 INFO:tasks.workunit.client.0.smithi129.stderr:58430: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1e 2024-03-25T23:11:23.314 INFO:tasks.workunit.client.0.smithi129.stderr:58430: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:23.314 INFO:tasks.workunit.client.0.smithi129.stderr:58430: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-25T23:11:23.314 INFO:tasks.workunit.client.0.smithi129.stderr:58430: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-25T23:11:23.314 INFO:tasks.workunit.client.0.smithi129.stderr:58430: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-25T23:11:23.315 INFO:tasks.workunit.client.0.smithi129.stderr:58430: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-25T23:11:23.315 INFO:tasks.workunit.client.0.smithi129.stderr:58430: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-25T23:11:23.315 INFO:tasks.workunit.client.0.smithi129.stderr:58430: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1e last_scrub_stamp 2024-03-25T23:11:23.315 INFO:tasks.workunit.client.0.smithi129.stderr:58430: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1e 2024-03-25T23:11:23.315 INFO:tasks.workunit.client.0.smithi129.stderr:58430: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:23.315 INFO:tasks.workunit.client.0.smithi129.stderr:58430: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2024-03-25T23:11:23.751 INFO:tasks.workunit.client.0.smithi129.stderr:58430: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: g58213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.4 2024-03-25T23:11:23.751 INFO:tasks.workunit.client.0.smithi129.stderr:58213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.4 2024-03-25T23:11:23.751 INFO:tasks.workunit.client.0.smithi129.stderr:58213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.4 2024-03-25T23:11:23.751 INFO:tasks.workunit.client.0.smithi129.stderr:58213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:23.751 INFO:tasks.workunit.client.0.smithi129.stderr:58213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:23.751 INFO:tasks.workunit.client.0.smithi129.stderr:58213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2024-03-25T23:11:23.752 INFO:tasks.workunit.client.0.smithi129.stderr:58213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:23.752 INFO:tasks.workunit.client.0.smithi129.stderr:58213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.4 query 2024-03-25T23:11:23.752 INFO:tasks.workunit.client.0.smithi129.stderr:58213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-25T23:11:23.752 INFO:tasks.workunit.client.0.smithi129.stderr:58213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-25T23:11:23.752 INFO:tasks.workunit.client.0.smithi129.stderr:58213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-25T23:11:23.752 INFO:tasks.workunit.client.0.smithi129.stderr:58213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-25T23:11:23.752 INFO:tasks.workunit.client.0.smithi129.stderr:58213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.4 2024-03-25T23:11:23.752 INFO:tasks.workunit.client.0.smithi129.stderr:58213: instructing pg 1.4 on osd.3 to scrub 2024-03-25T23:11:23.752 INFO:tasks.workunit.client.0.smithi129.stderr:58213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-25T23:11:23.752 INFO:tasks.workunit.client.0.smithi129.stderr:58213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.4") | .state' 2024-03-25T23:11:23.752 INFO:tasks.workunit.client.0.smithi129.stderr:58213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-25T23:11:23.752 INFO:tasks.workunit.client.0.smithi129.stderr:58213: dumped pgs 2024-03-25T23:11:23.752 INFO:tasks.workunit.client.0.smithi129.stderr:58213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.4 2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:23.752 INFO:tasks.workunit.client.0.smithi129.stderr:58213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.4 2024-03-25T23:11:23.753 INFO:tasks.workunit.client.0.smithi129.stderr:58213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:23.753 INFO:tasks.workunit.client.0.smithi129.stderr:58213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-25T23:11:23.753 INFO:tasks.workunit.client.0.smithi129.stderr:58213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-25T23:11:23.753 INFO:tasks.workunit.client.0.smithi129.stderr:58213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-25T23:11:23.753 INFO:tasks.workunit.client.0.smithi129.stderr:58213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-25T23:11:23.753 INFO:tasks.workunit.client.0.smithi129.stderr:58213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-25T23:11:23.753 INFO:tasks.workunit.client.0.smithi129.stderr:58213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.4 last_scrub_stamp 2024-03-25T23:11:23.753 INFO:tasks.workunit.client.0.smithi129.stderr:58213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.4 2024-03-25T23:11:23.753 INFO:tasks.workunit.client.0.smithi129.stderr:58213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:23.753 INFO:tasks.workunit.client.0.smithi129.stderr:58213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:23.753 INFO:tasks.workunit.client.0.smithi129.stderr:58213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2024-03-25T23:11:23.753 INFO:tasks.workunit.client.0.smithi129.stderr:58213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-25T23:09:44.508877+0000 '>' '' 2024-03-25T23:11:23.753 INFO:tasks.workunit.client.0.smithi129.stderr:58213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-25T23:11:23.753 INFO:tasks.workunit.client.0.smithi129.stderr:58213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-25T23:11:23.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:11:23.823 INFO:tasks.workunit.client.0.smithi129.stderr:58227: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.6 2024-03-25T23:11:23.823 INFO:tasks.workunit.client.0.smithi129.stderr:58227: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.6 2024-03-25T23:11:23.824 INFO:tasks.workunit.client.0.smithi129.stderr:58227: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.6 2024-03-25T23:11:23.824 INFO:tasks.workunit.client.0.smithi129.stderr:58227: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:23.824 INFO:tasks.workunit.client.0.smithi129.stderr:58227: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2024-03-25T23:11:23.824 INFO:tasks.workunit.client.0.smithi129.stderr:58227: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:23.824 INFO:tasks.workunit.client.0.smithi129.stderr:58227: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:23.824 INFO:tasks.workunit.client.0.smithi129.stderr:58227: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-25T23:11:23.824 INFO:tasks.workunit.client.0.smithi129.stderr:58227: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.6 query 2024-03-25T23:11:23.824 INFO:tasks.workunit.client.0.smithi129.stderr:58227: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-25T23:11:23.824 INFO:tasks.workunit.client.0.smithi129.stderr:58227: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-25T23:11:23.824 INFO:tasks.workunit.client.0.smithi129.stderr:58227: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-25T23:11:23.824 INFO:tasks.workunit.client.0.smithi129.stderr:58227: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.6 2024-03-25T23:11:23.824 INFO:tasks.workunit.client.0.smithi129.stderr:58227: instructing pg 1.6 on osd.6 to scrub 2024-03-25T23:11:23.824 INFO:tasks.workunit.client.0.smithi129.stderr:58227: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.6") | .state' 2024-03-25T23:11:23.824 INFO:tasks.workunit.client.0.smithi129.stderr:58227: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-25T23:11:23.825 INFO:tasks.workunit.client.0.smithi129.stderr:58227: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-25T23:11:23.825 INFO:tasks.workunit.client.0.smithi129.stderr:58227: dumped pgs 2024-03-25T23:11:23.825 INFO:tasks.workunit.client.0.smithi129.stderr:58227: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.6 2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:23.825 INFO:tasks.workunit.client.0.smithi129.stderr:58227: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.6 2024-03-25T23:11:23.825 INFO:tasks.workunit.client.0.smithi129.stderr:58227: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:23.825 INFO:tasks.workunit.client.0.smithi129.stderr:58227: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-25T23:11:23.825 INFO:tasks.workunit.client.0.smithi129.stderr:58227: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-25T23:11:23.825 INFO:tasks.workunit.client.0.smithi129.stderr:58227: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-25T23:11:23.825 INFO:tasks.workunit.client.0.smithi129.stderr:58227: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-25T23:11:23.825 INFO:tasks.workunit.client.0.smithi129.stderr:58227: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-25T23:11:23.825 INFO:tasks.workunit.client.0.smithi129.stderr:58227: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.6 last_scrub_stamp 2024-03-25T23:11:23.825 INFO:tasks.workunit.client.0.smithi129.stderr:58227: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.6 2024-03-25T23:11:23.825 INFO:tasks.workunit.client.0.smithi129.stderr:58227: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:23.825 INFO:tasks.workunit.client.0.smithi129.stderr:58227: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2024-03-25T23:11:23.826 INFO:tasks.workunit.client.0.smithi129.stderr:58227: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:23.826 INFO:tasks.workunit.client.0.smithi129.stderr:58227: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-25T23:09:44.508877+0000 '>' '' 2024-03-25T23:11:23.826 INFO:tasks.workunit.client.0.smithi129.stderr:58227: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-25T23:11:23.826 INFO:tasks.workunit.client.0.smithi129.stderr:58227: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-25T23:11:23.826 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:11:23.840 INFO:tasks.workunit.client.0.smithi129.stderr:58251: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.a 2024-03-25T23:11:23.840 INFO:tasks.workunit.client.0.smithi129.stderr:58251: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.a 2024-03-25T23:11:23.840 INFO:tasks.workunit.client.0.smithi129.stderr:58251: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.a 2024-03-25T23:11:23.840 INFO:tasks.workunit.client.0.smithi129.stderr:58251: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:23.840 INFO:tasks.workunit.client.0.smithi129.stderr:58251: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2024-03-25T23:11:23.841 INFO:tasks.workunit.client.0.smithi129.stderr:58251: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:23.841 INFO:tasks.workunit.client.0.smithi129.stderr:58251: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:23.841 INFO:tasks.workunit.client.0.smithi129.stderr:58251: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-25T23:11:23.841 INFO:tasks.workunit.client.0.smithi129.stderr:58251: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.a query 2024-03-25T23:11:23.841 INFO:tasks.workunit.client.0.smithi129.stderr:58251: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-25T23:11:23.841 INFO:tasks.workunit.client.0.smithi129.stderr:58251: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-25T23:11:23.841 INFO:tasks.workunit.client.0.smithi129.stderr:58251: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-25T23:11:23.841 INFO:tasks.workunit.client.0.smithi129.stderr:58251: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.a 2024-03-25T23:11:23.841 INFO:tasks.workunit.client.0.smithi129.stderr:58251: instructing pg 1.a on osd.1 to scrub 2024-03-25T23:11:23.841 INFO:tasks.workunit.client.0.smithi129.stderr:58251: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.a") | .state' 2024-03-25T23:11:23.841 INFO:tasks.workunit.client.0.smithi129.stderr:58251: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-25T23:11:23.841 INFO:tasks.workunit.client.0.smithi129.stderr:58251: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-25T23:11:23.841 INFO:tasks.workunit.client.0.smithi129.stderr:58251: dumped pgs 2024-03-25T23:11:23.841 INFO:tasks.workunit.client.0.smithi129.stderr:58251: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.a 2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:23.842 INFO:tasks.workunit.client.0.smithi129.stderr:58251: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.a 2024-03-25T23:11:23.842 INFO:tasks.workunit.client.0.smithi129.stderr:58251: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:23.842 INFO:tasks.workunit.client.0.smithi129.stderr:58251: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-25T23:11:23.842 INFO:tasks.workunit.client.0.smithi129.stderr:58251: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-25T23:11:23.842 INFO:tasks.workunit.client.0.smithi129.stderr:58251: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-25T23:11:23.842 INFO:tasks.workunit.client.0.smithi129.stderr:58251: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-25T23:11:23.842 INFO:tasks.workunit.client.0.smithi129.stderr:58251: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-25T23:11:23.842 INFO:tasks.workunit.client.0.smithi129.stderr:58251: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.a last_scrub_stamp 2024-03-25T23:11:23.842 INFO:tasks.workunit.client.0.smithi129.stderr:58251: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.a 2024-03-25T23:11:23.842 INFO:tasks.workunit.client.0.smithi129.stderr:58251: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:23.842 INFO:tasks.workunit.client.0.smithi129.stderr:58251: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2024-03-25T23:11:23.842 INFO:tasks.workunit.client.0.smithi129.stderr:58251: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:23.842 INFO:tasks.workunit.client.0.smithi129.stderr:58251: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-25T23:09:44.508877+0000 '>' '' 2024-03-25T23:11:23.842 INFO:tasks.workunit.client.0.smithi129.stderr:58251: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-25T23:11:23.843 INFO:tasks.workunit.client.0.smithi129.stderr:58251: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-25T23:11:23.843 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:11:24.095 INFO:tasks.workunit.client.0.smithi129.stderr:58206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.3 2024-03-25T23:11:24.096 INFO:tasks.workunit.client.0.smithi129.stderr:58206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.3 2024-03-25T23:11:24.096 INFO:tasks.workunit.client.0.smithi129.stderr:58206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-25T23:11:24.096 INFO:tasks.workunit.client.0.smithi129.stderr:58206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:24.096 INFO:tasks.workunit.client.0.smithi129.stderr:58206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:24.096 INFO:tasks.workunit.client.0.smithi129.stderr:58206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-25T23:11:24.096 INFO:tasks.workunit.client.0.smithi129.stderr:58206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.096 INFO:tasks.workunit.client.0.smithi129.stderr:58206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-25T23:11:24.096 INFO:tasks.workunit.client.0.smithi129.stderr:58206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.3 query 2024-03-25T23:11:24.096 INFO:tasks.workunit.client.0.smithi129.stderr:58206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-25T23:11:24.096 INFO:tasks.workunit.client.0.smithi129.stderr:58206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-25T23:11:24.096 INFO:tasks.workunit.client.0.smithi129.stderr:58206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-25T23:11:24.097 INFO:tasks.workunit.client.0.smithi129.stderr:58206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.3 2024-03-25T23:11:24.097 INFO:tasks.workunit.client.0.smithi129.stderr:58206: instructing pg 1.3 on osd.4 to scrub 2024-03-25T23:11:24.097 INFO:tasks.workunit.client.0.smithi129.stderr:58206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.3") | .state' 2024-03-25T23:11:24.097 INFO:tasks.workunit.client.0.smithi129.stderr:58206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-25T23:11:24.097 INFO:tasks.workunit.client.0.smithi129.stderr:58206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-25T23:11:24.097 INFO:tasks.workunit.client.0.smithi129.stderr:58206: dumped pgs 2024-03-25T23:11:24.097 INFO:tasks.workunit.client.0.smithi129.stderr:58206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.3 2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.097 INFO:tasks.workunit.client.0.smithi129.stderr:58206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.3 2024-03-25T23:11:24.097 INFO:tasks.workunit.client.0.smithi129.stderr:58206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.097 INFO:tasks.workunit.client.0.smithi129.stderr:58206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-25T23:11:24.097 INFO:tasks.workunit.client.0.smithi129.stderr:58206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-25T23:11:24.097 INFO:tasks.workunit.client.0.smithi129.stderr:58206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-25T23:11:24.097 INFO:tasks.workunit.client.0.smithi129.stderr:58206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-25T23:11:24.097 INFO:tasks.workunit.client.0.smithi129.stderr:58206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-25T23:11:24.098 INFO:tasks.workunit.client.0.smithi129.stderr:58206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-25T23:11:24.098 INFO:tasks.workunit.client.0.smithi129.stderr:58206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-25T23:11:24.098 INFO:tasks.workunit.client.0.smithi129.stderr:58206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:24.098 INFO:tasks.workunit.client.0.smithi129.stderr:58206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-25T23:11:24.098 INFO:tasks.workunit.client.0.smithi129.stderr:58206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:24.098 INFO:tasks.workunit.client.0.smithi129.stderr:58206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-25T23:09:44.508877+0000 '>' '' 2024-03-25T23:11:24.098 INFO:tasks.workunit.client.0.smithi129.stderr:58206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-25T23:11:24.098 INFO:tasks.workunit.client.0.smithi129.stderr:58206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-25T23:11:24.098 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:11:24.126 INFO:tasks.workunit.client.0.smithi129.stderr:58192: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1 2024-03-25T23:11:24.126 INFO:tasks.workunit.client.0.smithi129.stderr:58192: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1 2024-03-25T23:11:24.126 INFO:tasks.workunit.client.0.smithi129.stderr:58192: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-03-25T23:11:24.127 INFO:tasks.workunit.client.0.smithi129.stderr:58192: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:24.127 INFO:tasks.workunit.client.0.smithi129.stderr:58192: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:24.127 INFO:tasks.workunit.client.0.smithi129.stderr:58192: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-03-25T23:11:24.127 INFO:tasks.workunit.client.0.smithi129.stderr:58192: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.127 INFO:tasks.workunit.client.0.smithi129.stderr:58192: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-25T23:11:24.127 INFO:tasks.workunit.client.0.smithi129.stderr:58192: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1 query 2024-03-25T23:11:24.127 INFO:tasks.workunit.client.0.smithi129.stderr:58192: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-25T23:11:24.127 INFO:tasks.workunit.client.0.smithi129.stderr:58192: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-25T23:11:24.127 INFO:tasks.workunit.client.0.smithi129.stderr:58192: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-25T23:11:24.127 INFO:tasks.workunit.client.0.smithi129.stderr:58192: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1 2024-03-25T23:11:24.127 INFO:tasks.workunit.client.0.smithi129.stderr:58192: instructing pg 1.1 on osd.5 to scrub 2024-03-25T23:11:24.127 INFO:tasks.workunit.client.0.smithi129.stderr:58192: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1") | .state' 2024-03-25T23:11:24.128 INFO:tasks.workunit.client.0.smithi129.stderr:58192: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-25T23:11:24.128 INFO:tasks.workunit.client.0.smithi129.stderr:58192: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-25T23:11:24.128 INFO:tasks.workunit.client.0.smithi129.stderr:58192: dumped pgs 2024-03-25T23:11:24.128 INFO:tasks.workunit.client.0.smithi129.stderr:58192: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1 2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.128 INFO:tasks.workunit.client.0.smithi129.stderr:58192: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1 2024-03-25T23:11:24.128 INFO:tasks.workunit.client.0.smithi129.stderr:58192: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.128 INFO:tasks.workunit.client.0.smithi129.stderr:58192: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-25T23:11:24.128 INFO:tasks.workunit.client.0.smithi129.stderr:58192: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-25T23:11:24.128 INFO:tasks.workunit.client.0.smithi129.stderr:58192: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-25T23:11:24.128 INFO:tasks.workunit.client.0.smithi129.stderr:58192: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-25T23:11:24.128 INFO:tasks.workunit.client.0.smithi129.stderr:58192: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-25T23:11:24.128 INFO:tasks.workunit.client.0.smithi129.stderr:58192: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-03-25T23:11:24.128 INFO:tasks.workunit.client.0.smithi129.stderr:58192: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-03-25T23:11:24.128 INFO:tasks.workunit.client.0.smithi129.stderr:58192: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:24.129 INFO:tasks.workunit.client.0.smithi129.stderr:58192: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:24.129 INFO:tasks.workunit.client.0.smithi129.stderr:58192: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-03-25T23:11:24.129 INFO:tasks.workunit.client.0.smithi129.stderr:58192: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-25T23:09:44.508877+0000 '>' '' 2024-03-25T23:11:24.129 INFO:tasks.workunit.client.0.smithi129.stderr:58192: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-25T23:11:24.129 INFO:tasks.workunit.client.0.smithi129.stderr:58192: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-25T23:11:24.129 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:11:24.174 INFO:tasks.workunit.client.0.smithi129.stderr:58199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.2 2024-03-25T23:11:24.174 INFO:tasks.workunit.client.0.smithi129.stderr:58199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.2 2024-03-25T23:11:24.174 INFO:tasks.workunit.client.0.smithi129.stderr:58199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.2 2024-03-25T23:11:24.174 INFO:tasks.workunit.client.0.smithi129.stderr:58199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:24.174 INFO:tasks.workunit.client.0.smithi129.stderr:58199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:24.174 INFO:tasks.workunit.client.0.smithi129.stderr:58199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2024-03-25T23:11:24.174 INFO:tasks.workunit.client.0.smithi129.stderr:58199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.174 INFO:tasks.workunit.client.0.smithi129.stderr:58199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.2 query 2024-03-25T23:11:24.174 INFO:tasks.workunit.client.0.smithi129.stderr:58199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-25T23:11:24.174 INFO:tasks.workunit.client.0.smithi129.stderr:58199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-25T23:11:24.174 INFO:tasks.workunit.client.0.smithi129.stderr:58199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-25T23:11:24.175 INFO:tasks.workunit.client.0.smithi129.stderr:58199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-25T23:11:24.175 INFO:tasks.workunit.client.0.smithi129.stderr:58199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.2 2024-03-25T23:11:24.175 INFO:tasks.workunit.client.0.smithi129.stderr:58199: instructing pg 1.2 on osd.0 to scrub 2024-03-25T23:11:24.175 INFO:tasks.workunit.client.0.smithi129.stderr:58199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-25T23:11:24.175 INFO:tasks.workunit.client.0.smithi129.stderr:58199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.2") | .state' 2024-03-25T23:11:24.175 INFO:tasks.workunit.client.0.smithi129.stderr:58199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-25T23:11:24.175 INFO:tasks.workunit.client.0.smithi129.stderr:58199: dumped pgs 2024-03-25T23:11:24.175 INFO:tasks.workunit.client.0.smithi129.stderr:58199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.2 2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.175 INFO:tasks.workunit.client.0.smithi129.stderr:58199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.2 2024-03-25T23:11:24.175 INFO:tasks.workunit.client.0.smithi129.stderr:58199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.175 INFO:tasks.workunit.client.0.smithi129.stderr:58199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-25T23:11:24.175 INFO:tasks.workunit.client.0.smithi129.stderr:58199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-25T23:11:24.175 INFO:tasks.workunit.client.0.smithi129.stderr:58199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-25T23:11:24.175 INFO:tasks.workunit.client.0.smithi129.stderr:58199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-25T23:11:24.176 INFO:tasks.workunit.client.0.smithi129.stderr:58199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-25T23:11:24.176 INFO:tasks.workunit.client.0.smithi129.stderr:58199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.2 last_scrub_stamp 2024-03-25T23:11:24.176 INFO:tasks.workunit.client.0.smithi129.stderr:58199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.2 2024-03-25T23:11:24.176 INFO:tasks.workunit.client.0.smithi129.stderr:58199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:24.176 INFO:tasks.workunit.client.0.smithi129.stderr:58199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2024-03-25T23:11:24.176 INFO:tasks.workunit.client.0.smithi129.stderr:58199: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:24.176 INFO:tasks.workunit.client.0.smithi129.stderr:58199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-25T23:09:44.508877+0000 '>' '' 2024-03-25T23:11:24.176 INFO:tasks.workunit.client.0.smithi129.stderr:58199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-25T23:11:24.176 INFO:tasks.workunit.client.0.smithi129.stderr:58199: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-25T23:11:24.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:11:24.221 INFO:tasks.workunit.client.0.smithi129.stderr:58255: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.b 2024-03-25T23:11:24.221 INFO:tasks.workunit.client.0.smithi129.stderr:58255: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.b 2024-03-25T23:11:24.221 INFO:tasks.workunit.client.0.smithi129.stderr:58255: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.b 2024-03-25T23:11:24.221 INFO:tasks.workunit.client.0.smithi129.stderr:58255: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:24.222 INFO:tasks.workunit.client.0.smithi129.stderr:58255: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2024-03-25T23:11:24.222 INFO:tasks.workunit.client.0.smithi129.stderr:58255: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:24.222 INFO:tasks.workunit.client.0.smithi129.stderr:58255: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.222 INFO:tasks.workunit.client.0.smithi129.stderr:58255: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-25T23:11:24.222 INFO:tasks.workunit.client.0.smithi129.stderr:58255: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.b query 2024-03-25T23:11:24.222 INFO:tasks.workunit.client.0.smithi129.stderr:58255: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-25T23:11:24.222 INFO:tasks.workunit.client.0.smithi129.stderr:58255: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-25T23:11:24.222 INFO:tasks.workunit.client.0.smithi129.stderr:58255: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-25T23:11:24.222 INFO:tasks.workunit.client.0.smithi129.stderr:58255: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.b 2024-03-25T23:11:24.222 INFO:tasks.workunit.client.0.smithi129.stderr:58255: instructing pg 1.b on osd.1 to scrub 2024-03-25T23:11:24.222 INFO:tasks.workunit.client.0.smithi129.stderr:58255: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-25T23:11:24.222 INFO:tasks.workunit.client.0.smithi129.stderr:58255: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.b") | .state' 2024-03-25T23:11:24.222 INFO:tasks.workunit.client.0.smithi129.stderr:58255: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-25T23:11:24.223 INFO:tasks.workunit.client.0.smithi129.stderr:58255: dumped pgs 2024-03-25T23:11:24.223 INFO:tasks.workunit.client.0.smithi129.stderr:58255: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.b 2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.223 INFO:tasks.workunit.client.0.smithi129.stderr:58255: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.b 2024-03-25T23:11:24.223 INFO:tasks.workunit.client.0.smithi129.stderr:58255: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.223 INFO:tasks.workunit.client.0.smithi129.stderr:58255: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-25T23:11:24.223 INFO:tasks.workunit.client.0.smithi129.stderr:58255: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-25T23:11:24.223 INFO:tasks.workunit.client.0.smithi129.stderr:58255: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-25T23:11:24.223 INFO:tasks.workunit.client.0.smithi129.stderr:58255: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-25T23:11:24.223 INFO:tasks.workunit.client.0.smithi129.stderr:58255: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-25T23:11:24.223 INFO:tasks.workunit.client.0.smithi129.stderr:58255: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.b last_scrub_stamp 2024-03-25T23:11:24.223 INFO:tasks.workunit.client.0.smithi129.stderr:58255: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.b 2024-03-25T23:11:24.223 INFO:tasks.workunit.client.0.smithi129.stderr:58255: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:24.223 INFO:tasks.workunit.client.0.smithi129.stderr:58255: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:24.223 INFO:tasks.workunit.client.0.smithi129.stderr:58255: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2024-03-25T23:11:24.223 INFO:tasks.workunit.client.0.smithi129.stderr:58255: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-25T23:09:44.508877+0000 '>' '' 2024-03-25T23:11:24.224 INFO:tasks.workunit.client.0.smithi129.stderr:58255: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-25T23:11:24.224 INFO:tasks.workunit.client.0.smithi129.stderr:58255: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-25T23:11:24.224 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:11:24.302 INFO:tasks.workunit.client.0.smithi129.stderr:58229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.7 2024-03-25T23:11:24.302 INFO:tasks.workunit.client.0.smithi129.stderr:58229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.7 2024-03-25T23:11:24.302 INFO:tasks.workunit.client.0.smithi129.stderr:58229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.7 2024-03-25T23:11:24.302 INFO:tasks.workunit.client.0.smithi129.stderr:58229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:24.302 INFO:tasks.workunit.client.0.smithi129.stderr:58229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:24.302 INFO:tasks.workunit.client.0.smithi129.stderr:58229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2024-03-25T23:11:24.302 INFO:tasks.workunit.client.0.smithi129.stderr:58229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.302 INFO:tasks.workunit.client.0.smithi129.stderr:58229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.7 query 2024-03-25T23:11:24.303 INFO:tasks.workunit.client.0.smithi129.stderr:58229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-25T23:11:24.303 INFO:tasks.workunit.client.0.smithi129.stderr:58229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-25T23:11:24.303 INFO:tasks.workunit.client.0.smithi129.stderr:58229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-25T23:11:24.303 INFO:tasks.workunit.client.0.smithi129.stderr:58229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-25T23:11:24.303 INFO:tasks.workunit.client.0.smithi129.stderr:58229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.7 2024-03-25T23:11:24.303 INFO:tasks.workunit.client.0.smithi129.stderr:58229: instructing pg 1.7 on osd.6 to scrub 2024-03-25T23:11:24.303 INFO:tasks.workunit.client.0.smithi129.stderr:58229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-25T23:11:24.303 INFO:tasks.workunit.client.0.smithi129.stderr:58229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.7") | .state' 2024-03-25T23:11:24.303 INFO:tasks.workunit.client.0.smithi129.stderr:58229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-25T23:11:24.303 INFO:tasks.workunit.client.0.smithi129.stderr:58229: dumped pgs 2024-03-25T23:11:24.303 INFO:tasks.workunit.client.0.smithi129.stderr:58229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.7 2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.303 INFO:tasks.workunit.client.0.smithi129.stderr:58229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.7 2024-03-25T23:11:24.303 INFO:tasks.workunit.client.0.smithi129.stderr:58229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.303 INFO:tasks.workunit.client.0.smithi129.stderr:58229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-25T23:11:24.304 INFO:tasks.workunit.client.0.smithi129.stderr:58229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-25T23:11:24.304 INFO:tasks.workunit.client.0.smithi129.stderr:58229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-25T23:11:24.304 INFO:tasks.workunit.client.0.smithi129.stderr:58229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-25T23:11:24.304 INFO:tasks.workunit.client.0.smithi129.stderr:58229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-25T23:11:24.304 INFO:tasks.workunit.client.0.smithi129.stderr:58229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.7 last_scrub_stamp 2024-03-25T23:11:24.304 INFO:tasks.workunit.client.0.smithi129.stderr:58229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.7 2024-03-25T23:11:24.304 INFO:tasks.workunit.client.0.smithi129.stderr:58229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:24.304 INFO:tasks.workunit.client.0.smithi129.stderr:58229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:24.304 INFO:tasks.workunit.client.0.smithi129.stderr:58229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2024-03-25T23:11:24.304 INFO:tasks.workunit.client.0.smithi129.stderr:58229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-25T23:09:44.508877+0000 '>' '' 2024-03-25T23:11:24.304 INFO:tasks.workunit.client.0.smithi129.stderr:58229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-25T23:11:24.304 INFO:tasks.workunit.client.0.smithi129.stderr:58229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-25T23:11:24.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:11:24.385 INFO:tasks.workunit.client.0.smithi129.stderr:58278: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.d 2024-03-25T23:11:24.385 INFO:tasks.workunit.client.0.smithi129.stderr:58278: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.d 2024-03-25T23:11:24.385 INFO:tasks.workunit.client.0.smithi129.stderr:58278: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.d 2024-03-25T23:11:24.385 INFO:tasks.workunit.client.0.smithi129.stderr:58278: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:24.385 INFO:tasks.workunit.client.0.smithi129.stderr:58278: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:24.385 INFO:tasks.workunit.client.0.smithi129.stderr:58278: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2024-03-25T23:11:24.385 INFO:tasks.workunit.client.0.smithi129.stderr:58278: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.385 INFO:tasks.workunit.client.0.smithi129.stderr:58278: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-25T23:11:24.386 INFO:tasks.workunit.client.0.smithi129.stderr:58278: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.d query 2024-03-25T23:11:24.386 INFO:tasks.workunit.client.0.smithi129.stderr:58278: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-25T23:11:24.386 INFO:tasks.workunit.client.0.smithi129.stderr:58278: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-25T23:11:24.386 INFO:tasks.workunit.client.0.smithi129.stderr:58278: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-25T23:11:24.386 INFO:tasks.workunit.client.0.smithi129.stderr:58278: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.d 2024-03-25T23:11:24.386 INFO:tasks.workunit.client.0.smithi129.stderr:58278: instructing pg 1.d on osd.7 to scrub 2024-03-25T23:11:24.386 INFO:tasks.workunit.client.0.smithi129.stderr:58278: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-25T23:11:24.386 INFO:tasks.workunit.client.0.smithi129.stderr:58278: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.d") | .state' 2024-03-25T23:11:24.386 INFO:tasks.workunit.client.0.smithi129.stderr:58278: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-25T23:11:24.386 INFO:tasks.workunit.client.0.smithi129.stderr:58278: dumped pgs 2024-03-25T23:11:24.386 INFO:tasks.workunit.client.0.smithi129.stderr:58278: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:198: pg_scrub_mod: recovery=true 2024-03-25T23:11:24.386 INFO:tasks.workunit.client.0.smithi129.stderr:58278: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.d 2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.386 INFO:tasks.workunit.client.0.smithi129.stderr:58278: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.d 2024-03-25T23:11:24.386 INFO:tasks.workunit.client.0.smithi129.stderr:58278: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.387 INFO:tasks.workunit.client.0.smithi129.stderr:58278: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-25T23:11:24.387 INFO:tasks.workunit.client.0.smithi129.stderr:58278: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-25T23:11:24.387 INFO:tasks.workunit.client.0.smithi129.stderr:58278: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-25T23:11:24.387 INFO:tasks.workunit.client.0.smithi129.stderr:58278: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-25T23:11:24.387 INFO:tasks.workunit.client.0.smithi129.stderr:58278: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-25T23:11:24.387 INFO:tasks.workunit.client.0.smithi129.stderr:58278: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.d last_scrub_stamp 2024-03-25T23:11:24.387 INFO:tasks.workunit.client.0.smithi129.stderr:58278: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.d 2024-03-25T23:11:24.387 INFO:tasks.workunit.client.0.smithi129.stderr:58278: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:24.388 INFO:tasks.workunit.client.0.smithi129.stderr:58278: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:24.446 INFO:tasks.workunit.client.0.smithi129.stderr:58278: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | se58265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.c 2024-03-25T23:11:24.447 INFO:tasks.workunit.client.0.smithi129.stderr:58265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.c 2024-03-25T23:11:24.447 INFO:tasks.workunit.client.0.smithi129.stderr:58265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.c 2024-03-25T23:11:24.447 INFO:tasks.workunit.client.0.smithi129.stderr:58265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:24.447 INFO:tasks.workunit.client.0.smithi129.stderr:58265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:24.447 INFO:tasks.workunit.client.0.smithi129.stderr:58265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2024-03-25T23:11:24.447 INFO:tasks.workunit.client.0.smithi129.stderr:58265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.447 INFO:tasks.workunit.client.0.smithi129.stderr:58265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-25T23:11:24.447 INFO:tasks.workunit.client.0.smithi129.stderr:58265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.c query 2024-03-25T23:11:24.447 INFO:tasks.workunit.client.0.smithi129.stderr:58265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-25T23:11:24.447 INFO:tasks.workunit.client.0.smithi129.stderr:58265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-25T23:11:24.447 INFO:tasks.workunit.client.0.smithi129.stderr:58265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-25T23:11:24.447 INFO:tasks.workunit.client.0.smithi129.stderr:58265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.c 2024-03-25T23:11:24.448 INFO:tasks.workunit.client.0.smithi129.stderr:58265: instructing pg 1.c on osd.3 to scrub 2024-03-25T23:11:24.448 INFO:tasks.workunit.client.0.smithi129.stderr:58265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-25T23:11:24.448 INFO:tasks.workunit.client.0.smithi129.stderr:58265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.c") | .state' 2024-03-25T23:11:24.448 INFO:tasks.workunit.client.0.smithi129.stderr:58265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-25T23:11:24.448 INFO:tasks.workunit.client.0.smithi129.stderr:58265: dumped pgs 2024-03-25T23:11:24.448 INFO:tasks.workunit.client.0.smithi129.stderr:58265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.c 2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.448 INFO:tasks.workunit.client.0.smithi129.stderr:58265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.c 2024-03-25T23:11:24.448 INFO:tasks.workunit.client.0.smithi129.stderr:58265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.448 INFO:tasks.workunit.client.0.smithi129.stderr:58265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-25T23:11:24.448 INFO:tasks.workunit.client.0.smithi129.stderr:58265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-25T23:11:24.448 INFO:tasks.workunit.client.0.smithi129.stderr:58265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-25T23:11:24.448 INFO:tasks.workunit.client.0.smithi129.stderr:58265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-25T23:11:24.448 INFO:tasks.workunit.client.0.smithi129.stderr:58265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-25T23:11:24.449 INFO:tasks.workunit.client.0.smithi129.stderr:58265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.c last_scrub_stamp 2024-03-25T23:11:24.449 INFO:tasks.workunit.client.0.smithi129.stderr:58265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.c 2024-03-25T23:11:24.449 INFO:tasks.workunit.client.0.smithi129.stderr:58265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:24.449 INFO:tasks.workunit.client.0.smithi129.stderr:58265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2024-03-25T23:11:24.449 INFO:tasks.workunit.client.0.smithi129.stderr:58265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:24.449 INFO:tasks.workunit.client.0.smithi129.stderr:58265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-25T23:11:20.715695+0000 '>' '' 2024-03-25T23:11:24.449 INFO:tasks.workunit.client.0.smithi129.stderr:58265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-25T23:11:24.449 INFO:tasks.workunit.client.0.smithi129.stderr:58265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-25T23:11:24.449 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:11:24.493 INFO:tasks.workunit.client.0.smithi129.stderr:58356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.18 2024-03-25T23:11:24.493 INFO:tasks.workunit.client.0.smithi129.stderr:58356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.18 2024-03-25T23:11:24.493 INFO:tasks.workunit.client.0.smithi129.stderr:58356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.18 2024-03-25T23:11:24.493 INFO:tasks.workunit.client.0.smithi129.stderr:58356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:24.493 INFO:tasks.workunit.client.0.smithi129.stderr:58356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:24.493 INFO:tasks.workunit.client.0.smithi129.stderr:58356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2024-03-25T23:11:24.493 INFO:tasks.workunit.client.0.smithi129.stderr:58356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.493 INFO:tasks.workunit.client.0.smithi129.stderr:58356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-25T23:11:24.493 INFO:tasks.workunit.client.0.smithi129.stderr:58356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.18 query 2024-03-25T23:11:24.493 INFO:tasks.workunit.client.0.smithi129.stderr:58356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-25T23:11:24.493 INFO:tasks.workunit.client.0.smithi129.stderr:58356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-25T23:11:24.493 INFO:tasks.workunit.client.0.smithi129.stderr:58356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-25T23:11:24.494 INFO:tasks.workunit.client.0.smithi129.stderr:58356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.18 2024-03-25T23:11:24.494 INFO:tasks.workunit.client.0.smithi129.stderr:58356: instructing pg 1.18 on osd.5 to scrub 2024-03-25T23:11:24.494 INFO:tasks.workunit.client.0.smithi129.stderr:58356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-25T23:11:24.494 INFO:tasks.workunit.client.0.smithi129.stderr:58356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-25T23:11:24.494 INFO:tasks.workunit.client.0.smithi129.stderr:58356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.18") | .state' 2024-03-25T23:11:24.494 INFO:tasks.workunit.client.0.smithi129.stderr:58356: dumped pgs 2024-03-25T23:11:24.494 INFO:tasks.workunit.client.0.smithi129.stderr:58356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.18 2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.494 INFO:tasks.workunit.client.0.smithi129.stderr:58356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.18 2024-03-25T23:11:24.494 INFO:tasks.workunit.client.0.smithi129.stderr:58356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.494 INFO:tasks.workunit.client.0.smithi129.stderr:58356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-25T23:11:24.494 INFO:tasks.workunit.client.0.smithi129.stderr:58356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-25T23:11:24.494 INFO:tasks.workunit.client.0.smithi129.stderr:58356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-25T23:11:24.495 INFO:tasks.workunit.client.0.smithi129.stderr:58356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-25T23:11:24.495 INFO:tasks.workunit.client.0.smithi129.stderr:58356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-25T23:11:24.495 INFO:tasks.workunit.client.0.smithi129.stderr:58356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.18 last_scrub_stamp 2024-03-25T23:11:24.495 INFO:tasks.workunit.client.0.smithi129.stderr:58356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.18 2024-03-25T23:11:24.495 INFO:tasks.workunit.client.0.smithi129.stderr:58356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:24.495 INFO:tasks.workunit.client.0.smithi129.stderr:58356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2024-03-25T23:11:24.495 INFO:tasks.workunit.client.0.smithi129.stderr:58356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:24.495 INFO:tasks.workunit.client.0.smithi129.stderr:58356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-25T23:09:44.508877+0000 '>' '' 2024-03-25T23:11:24.495 INFO:tasks.workunit.client.0.smithi129.stderr:58356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-25T23:11:24.495 INFO:tasks.workunit.client.0.smithi129.stderr:58356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-25T23:11:24.500 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:11:24.520 INFO:tasks.workunit.client.0.smithi129.stderr:58237: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.9 2024-03-25T23:11:24.520 INFO:tasks.workunit.client.0.smithi129.stderr:58237: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.9 2024-03-25T23:11:24.520 INFO:tasks.workunit.client.0.smithi129.stderr:58237: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.9 2024-03-25T23:11:24.520 INFO:tasks.workunit.client.0.smithi129.stderr:58237: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:24.520 INFO:tasks.workunit.client.0.smithi129.stderr:58237: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2024-03-25T23:11:24.520 INFO:tasks.workunit.client.0.smithi129.stderr:58237: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:24.520 INFO:tasks.workunit.client.0.smithi129.stderr:58237: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.521 INFO:tasks.workunit.client.0.smithi129.stderr:58237: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.9 query 2024-03-25T23:11:24.521 INFO:tasks.workunit.client.0.smithi129.stderr:58237: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-25T23:11:24.521 INFO:tasks.workunit.client.0.smithi129.stderr:58237: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-25T23:11:24.521 INFO:tasks.workunit.client.0.smithi129.stderr:58237: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-25T23:11:24.521 INFO:tasks.workunit.client.0.smithi129.stderr:58237: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-25T23:11:24.521 INFO:tasks.workunit.client.0.smithi129.stderr:58237: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.9 2024-03-25T23:11:24.521 INFO:tasks.workunit.client.0.smithi129.stderr:58237: instructing pg 1.9 on osd.1 to scrub 2024-03-25T23:11:24.521 INFO:tasks.workunit.client.0.smithi129.stderr:58237: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-25T23:11:24.521 INFO:tasks.workunit.client.0.smithi129.stderr:58237: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.9") | .state' 2024-03-25T23:11:24.521 INFO:tasks.workunit.client.0.smithi129.stderr:58237: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-25T23:11:24.521 INFO:tasks.workunit.client.0.smithi129.stderr:58237: dumped pgs 2024-03-25T23:11:24.521 INFO:tasks.workunit.client.0.smithi129.stderr:58237: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.9 2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.521 INFO:tasks.workunit.client.0.smithi129.stderr:58237: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.9 2024-03-25T23:11:24.521 INFO:tasks.workunit.client.0.smithi129.stderr:58237: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.522 INFO:tasks.workunit.client.0.smithi129.stderr:58237: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-25T23:11:24.522 INFO:tasks.workunit.client.0.smithi129.stderr:58237: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-25T23:11:24.522 INFO:tasks.workunit.client.0.smithi129.stderr:58237: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-25T23:11:24.522 INFO:tasks.workunit.client.0.smithi129.stderr:58237: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-25T23:11:24.522 INFO:tasks.workunit.client.0.smithi129.stderr:58237: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-25T23:11:24.522 INFO:tasks.workunit.client.0.smithi129.stderr:58237: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.9 last_scrub_stamp 2024-03-25T23:11:24.522 INFO:tasks.workunit.client.0.smithi129.stderr:58237: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.9 2024-03-25T23:11:24.522 INFO:tasks.workunit.client.0.smithi129.stderr:58237: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:24.522 INFO:tasks.workunit.client.0.smithi129.stderr:58237: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:24.522 INFO:tasks.workunit.client.0.smithi129.stderr:58237: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2024-03-25T23:11:24.522 INFO:tasks.workunit.client.0.smithi129.stderr:58237: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-25T23:11:20.629501+0000 '>' '' 2024-03-25T23:11:24.522 INFO:tasks.workunit.client.0.smithi129.stderr:58237: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-25T23:11:24.522 INFO:tasks.workunit.client.0.smithi129.stderr:58237: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-25T23:11:24.522 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:11:24.575 INFO:tasks.workunit.client.0.smithi129.stderr:58185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.0 2024-03-25T23:11:24.575 INFO:tasks.workunit.client.0.smithi129.stderr:58185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.0 2024-03-25T23:11:24.575 INFO:tasks.workunit.client.0.smithi129.stderr:58185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-25T23:11:24.575 INFO:tasks.workunit.client.0.smithi129.stderr:58185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:24.575 INFO:tasks.workunit.client.0.smithi129.stderr:58185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:24.575 INFO:tasks.workunit.client.0.smithi129.stderr:58185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-25T23:11:24.575 INFO:tasks.workunit.client.0.smithi129.stderr:58185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.575 INFO:tasks.workunit.client.0.smithi129.stderr:58185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.0 query 2024-03-25T23:11:24.575 INFO:tasks.workunit.client.0.smithi129.stderr:58185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-25T23:11:24.575 INFO:tasks.workunit.client.0.smithi129.stderr:58185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-25T23:11:24.575 INFO:tasks.workunit.client.0.smithi129.stderr:58185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-25T23:11:24.576 INFO:tasks.workunit.client.0.smithi129.stderr:58185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-25T23:11:24.576 INFO:tasks.workunit.client.0.smithi129.stderr:58185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.0 2024-03-25T23:11:24.576 INFO:tasks.workunit.client.0.smithi129.stderr:58185: instructing pg 1.0 on osd.7 to scrub 2024-03-25T23:11:24.576 INFO:tasks.workunit.client.0.smithi129.stderr:58185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-25T23:11:24.576 INFO:tasks.workunit.client.0.smithi129.stderr:58185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.0") | .state' 2024-03-25T23:11:24.576 INFO:tasks.workunit.client.0.smithi129.stderr:58185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-25T23:11:24.576 INFO:tasks.workunit.client.0.smithi129.stderr:58185: dumped pgs 2024-03-25T23:11:24.576 INFO:tasks.workunit.client.0.smithi129.stderr:58185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.0 2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.576 INFO:tasks.workunit.client.0.smithi129.stderr:58185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.0 2024-03-25T23:11:24.576 INFO:tasks.workunit.client.0.smithi129.stderr:58185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.576 INFO:tasks.workunit.client.0.smithi129.stderr:58185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-25T23:11:24.576 INFO:tasks.workunit.client.0.smithi129.stderr:58185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-25T23:11:24.577 INFO:tasks.workunit.client.0.smithi129.stderr:58185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-25T23:11:24.577 INFO:tasks.workunit.client.0.smithi129.stderr:58185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-25T23:11:24.577 INFO:tasks.workunit.client.0.smithi129.stderr:58185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-25T23:11:24.577 INFO:tasks.workunit.client.0.smithi129.stderr:58185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-25T23:11:24.577 INFO:tasks.workunit.client.0.smithi129.stderr:58185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-25T23:11:24.577 INFO:tasks.workunit.client.0.smithi129.stderr:58185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:24.577 INFO:tasks.workunit.client.0.smithi129.stderr:58185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:24.577 INFO:tasks.workunit.client.0.smithi129.stderr:58185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-25T23:11:24.577 INFO:tasks.workunit.client.0.smithi129.stderr:58185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-25T23:09:44.508877+0000 '>' '' 2024-03-25T23:11:24.577 INFO:tasks.workunit.client.0.smithi129.stderr:58185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-25T23:11:24.577 INFO:tasks.workunit.client.0.smithi129.stderr:58185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-25T23:11:24.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:11:24.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-25T23:11:24.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-25T23:11:24.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-25T23:11:24.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-25T23:11:24.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58187 2024-03-25T23:11:24.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-25T23:11:24.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-25T23:11:24.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-25T23:11:24.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-25T23:11:24.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58195 2024-03-25T23:11:24.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-25T23:11:24.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-25T23:11:24.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-25T23:11:24.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-25T23:11:24.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58201 2024-03-25T23:11:24.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-25T23:11:24.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-25T23:11:24.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-25T23:11:24.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-25T23:11:24.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58207 2024-03-25T23:11:24.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-25T23:11:24.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-25T23:11:24.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-25T23:11:24.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-25T23:11:24.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58215 2024-03-25T23:11:24.581 INFO:tasks.workunit.client.0.smithi129.stderr:58344: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.16 2024-03-25T23:11:24.581 INFO:tasks.workunit.client.0.smithi129.stderr:58344: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.16 2024-03-25T23:11:24.581 INFO:tasks.workunit.client.0.smithi129.stderr:58344: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.16 2024-03-25T23:11:24.581 INFO:tasks.workunit.client.0.smithi129.stderr:58344: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:24.581 INFO:tasks.workunit.client.0.smithi129.stderr:58344: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:24.581 INFO:tasks.workunit.client.0.smithi129.stderr:58344: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2024-03-25T23:11:24.581 INFO:tasks.workunit.client.0.smithi129.stderr:58344: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.581 INFO:tasks.workunit.client.0.smithi129.stderr:58344: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.16 query 2024-03-25T23:11:24.581 INFO:tasks.workunit.client.0.smithi129.stderr:58344: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-25T23:11:24.581 INFO:tasks.workunit.client.0.smithi129.stderr:58344: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-25T23:11:24.581 INFO:tasks.workunit.client.0.smithi129.stderr:58344: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-25T23:11:24.581 INFO:tasks.workunit.client.0.smithi129.stderr:58344: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-25T23:11:24.582 INFO:tasks.workunit.client.0.smithi129.stderr:58344: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.16 2024-03-25T23:11:24.582 INFO:tasks.workunit.client.0.smithi129.stderr:58344: instructing pg 1.16 on osd.6 to scrub 2024-03-25T23:11:24.582 INFO:tasks.workunit.client.0.smithi129.stderr:58344: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-25T23:11:24.582 INFO:tasks.workunit.client.0.smithi129.stderr:58344: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.16") | .state' 2024-03-25T23:11:24.582 INFO:tasks.workunit.client.0.smithi129.stderr:58344: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-25T23:11:24.582 INFO:tasks.workunit.client.0.smithi129.stderr:58344: dumped pgs 2024-03-25T23:11:24.582 INFO:tasks.workunit.client.0.smithi129.stderr:58344: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.16 2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.582 INFO:tasks.workunit.client.0.smithi129.stderr:58344: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.16 2024-03-25T23:11:24.582 INFO:tasks.workunit.client.0.smithi129.stderr:58344: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.582 INFO:tasks.workunit.client.0.smithi129.stderr:58344: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-25T23:11:24.582 INFO:tasks.workunit.client.0.smithi129.stderr:58344: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-25T23:11:24.582 INFO:tasks.workunit.client.0.smithi129.stderr:58344: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-25T23:11:24.582 INFO:tasks.workunit.client.0.smithi129.stderr:58344: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-25T23:11:24.582 INFO:tasks.workunit.client.0.smithi129.stderr:58344: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-25T23:11:24.583 INFO:tasks.workunit.client.0.smithi129.stderr:58344: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.16 last_scrub_stamp 2024-03-25T23:11:24.583 INFO:tasks.workunit.client.0.smithi129.stderr:58344: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.16 2024-03-25T23:11:24.583 INFO:tasks.workunit.client.0.smithi129.stderr:58344: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:24.583 INFO:tasks.workunit.client.0.smithi129.stderr:58344: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2024-03-25T23:11:24.583 INFO:tasks.workunit.client.0.smithi129.stderr:58344: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:24.583 INFO:tasks.workunit.client.0.smithi129.stderr:58344: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-25T23:11:20.183697+0000 '>' '' 2024-03-25T23:11:24.583 INFO:tasks.workunit.client.0.smithi129.stderr:58344: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-25T23:11:24.583 INFO:tasks.workunit.client.0.smithi129.stderr:58344: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-25T23:11:24.583 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:11:24.583 INFO:tasks.workunit.client.0.smithi129.stderr:58409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1c 2024-03-25T23:11:24.584 INFO:tasks.workunit.client.0.smithi129.stderr:58409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1c 2024-03-25T23:11:24.584 INFO:tasks.workunit.client.0.smithi129.stderr:58409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1c 2024-03-25T23:11:24.584 INFO:tasks.workunit.client.0.smithi129.stderr:58409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:24.584 INFO:tasks.workunit.client.0.smithi129.stderr:58409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:24.584 INFO:tasks.workunit.client.0.smithi129.stderr:58409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2024-03-25T23:11:24.584 INFO:tasks.workunit.client.0.smithi129.stderr:58409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.584 INFO:tasks.workunit.client.0.smithi129.stderr:58409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1c query 2024-03-25T23:11:24.584 INFO:tasks.workunit.client.0.smithi129.stderr:58409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-25T23:11:24.584 INFO:tasks.workunit.client.0.smithi129.stderr:58409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-25T23:11:24.584 INFO:tasks.workunit.client.0.smithi129.stderr:58409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-25T23:11:24.584 INFO:tasks.workunit.client.0.smithi129.stderr:58409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-25T23:11:24.584 INFO:tasks.workunit.client.0.smithi129.stderr:58409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1c 2024-03-25T23:11:24.584 INFO:tasks.workunit.client.0.smithi129.stderr:58409: instructing pg 1.1c on osd.0 to scrub 2024-03-25T23:11:24.584 INFO:tasks.workunit.client.0.smithi129.stderr:58409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-25T23:11:24.585 INFO:tasks.workunit.client.0.smithi129.stderr:58409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-25T23:11:24.585 INFO:tasks.workunit.client.0.smithi129.stderr:58409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1c") | .state' 2024-03-25T23:11:24.585 INFO:tasks.workunit.client.0.smithi129.stderr:58409: dumped pgs 2024-03-25T23:11:24.585 INFO:tasks.workunit.client.0.smithi129.stderr:58409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1c 2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.585 INFO:tasks.workunit.client.0.smithi129.stderr:58409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1c 2024-03-25T23:11:24.585 INFO:tasks.workunit.client.0.smithi129.stderr:58409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.585 INFO:tasks.workunit.client.0.smithi129.stderr:58409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-25T23:11:24.585 INFO:tasks.workunit.client.0.smithi129.stderr:58409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-25T23:11:24.585 INFO:tasks.workunit.client.0.smithi129.stderr:58409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-25T23:11:24.585 INFO:tasks.workunit.client.0.smithi129.stderr:58409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-25T23:11:24.585 INFO:tasks.workunit.client.0.smithi129.stderr:58409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-25T23:11:24.585 INFO:tasks.workunit.client.0.smithi129.stderr:58409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1c last_scrub_stamp 2024-03-25T23:11:24.585 INFO:tasks.workunit.client.0.smithi129.stderr:58409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1c 2024-03-25T23:11:24.585 INFO:tasks.workunit.client.0.smithi129.stderr:58409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:24.585 INFO:tasks.workunit.client.0.smithi129.stderr:58409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2024-03-25T23:11:24.586 INFO:tasks.workunit.client.0.smithi129.stderr:58409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:24.586 INFO:tasks.workunit.client.0.smithi129.stderr:58409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-25T23:09:44.508877+0000 '>' '' 2024-03-25T23:11:24.586 INFO:tasks.workunit.client.0.smithi129.stderr:58409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-25T23:11:24.586 INFO:tasks.workunit.client.0.smithi129.stderr:58409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-25T23:11:24.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:11:24.592 INFO:tasks.workunit.client.0.smithi129.stderr:58220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.5 2024-03-25T23:11:24.592 INFO:tasks.workunit.client.0.smithi129.stderr:58220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.5 2024-03-25T23:11:24.592 INFO:tasks.workunit.client.0.smithi129.stderr:58220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.5 2024-03-25T23:11:24.592 INFO:tasks.workunit.client.0.smithi129.stderr:58220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:24.592 INFO:tasks.workunit.client.0.smithi129.stderr:58220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2024-03-25T23:11:24.592 INFO:tasks.workunit.client.0.smithi129.stderr:58220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:24.592 INFO:tasks.workunit.client.0.smithi129.stderr:58220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.592 INFO:tasks.workunit.client.0.smithi129.stderr:58220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.5 query 2024-03-25T23:11:24.593 INFO:tasks.workunit.client.0.smithi129.stderr:58220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-25T23:11:24.593 INFO:tasks.workunit.client.0.smithi129.stderr:58220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-25T23:11:24.593 INFO:tasks.workunit.client.0.smithi129.stderr:58220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-25T23:11:24.593 INFO:tasks.workunit.client.0.smithi129.stderr:58220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-25T23:11:24.593 INFO:tasks.workunit.client.0.smithi129.stderr:58220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.5 2024-03-25T23:11:24.593 INFO:tasks.workunit.client.0.smithi129.stderr:58220: instructing pg 1.5 on osd.4 to scrub 2024-03-25T23:11:24.593 INFO:tasks.workunit.client.0.smithi129.stderr:58220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-25T23:11:24.593 INFO:tasks.workunit.client.0.smithi129.stderr:58220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.5") | .state' 2024-03-25T23:11:24.593 INFO:tasks.workunit.client.0.smithi129.stderr:58220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-25T23:11:24.593 INFO:tasks.workunit.client.0.smithi129.stderr:58220: dumped pgs 2024-03-25T23:11:24.593 INFO:tasks.workunit.client.0.smithi129.stderr:58220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.5 2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.593 INFO:tasks.workunit.client.0.smithi129.stderr:58220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.5 2024-03-25T23:11:24.593 INFO:tasks.workunit.client.0.smithi129.stderr:58220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.593 INFO:tasks.workunit.client.0.smithi129.stderr:58220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-25T23:11:24.594 INFO:tasks.workunit.client.0.smithi129.stderr:58220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-25T23:11:24.594 INFO:tasks.workunit.client.0.smithi129.stderr:58220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-25T23:11:24.594 INFO:tasks.workunit.client.0.smithi129.stderr:58220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-25T23:11:24.594 INFO:tasks.workunit.client.0.smithi129.stderr:58220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-25T23:11:24.594 INFO:tasks.workunit.client.0.smithi129.stderr:58220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.5 last_scrub_stamp 2024-03-25T23:11:24.594 INFO:tasks.workunit.client.0.smithi129.stderr:58220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.5 2024-03-25T23:11:24.594 INFO:tasks.workunit.client.0.smithi129.stderr:58220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:24.594 INFO:tasks.workunit.client.0.smithi129.stderr:58220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:24.594 INFO:tasks.workunit.client.0.smithi129.stderr:58220: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2024-03-25T23:11:24.594 INFO:tasks.workunit.client.0.smithi129.stderr:58220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-25T23:09:44.508877+0000 '>' '' 2024-03-25T23:11:24.594 INFO:tasks.workunit.client.0.smithi129.stderr:58220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-25T23:11:24.594 INFO:tasks.workunit.client.0.smithi129.stderr:58220: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-25T23:11:24.595 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:11:24.595 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-25T23:11:24.595 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-25T23:11:24.595 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-25T23:11:24.595 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-25T23:11:24.595 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58217 2024-03-25T23:11:24.595 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-25T23:11:24.595 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-25T23:11:24.595 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-25T23:11:24.595 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-25T23:11:24.595 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58223 2024-03-25T23:11:24.595 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-25T23:11:24.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-25T23:11:24.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-25T23:11:24.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-25T23:11:24.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58230 2024-03-25T23:11:24.688 INFO:tasks.workunit.client.0.smithi129.stderr:58284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.f 2024-03-25T23:11:24.688 INFO:tasks.workunit.client.0.smithi129.stderr:58284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.f 2024-03-25T23:11:24.688 INFO:tasks.workunit.client.0.smithi129.stderr:58284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.f 2024-03-25T23:11:24.688 INFO:tasks.workunit.client.0.smithi129.stderr:58284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:24.688 INFO:tasks.workunit.client.0.smithi129.stderr:58284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:24.688 INFO:tasks.workunit.client.0.smithi129.stderr:58284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2024-03-25T23:11:24.688 INFO:tasks.workunit.client.0.smithi129.stderr:58284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.688 INFO:tasks.workunit.client.0.smithi129.stderr:58284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-25T23:11:24.688 INFO:tasks.workunit.client.0.smithi129.stderr:58284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.f query 2024-03-25T23:11:24.689 INFO:tasks.workunit.client.0.smithi129.stderr:58284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-25T23:11:24.689 INFO:tasks.workunit.client.0.smithi129.stderr:58284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-25T23:11:24.689 INFO:tasks.workunit.client.0.smithi129.stderr:58284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-25T23:11:24.689 INFO:tasks.workunit.client.0.smithi129.stderr:58284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.f 2024-03-25T23:11:24.689 INFO:tasks.workunit.client.0.smithi129.stderr:58284: instructing pg 1.f on osd.6 to scrub 2024-03-25T23:11:24.689 INFO:tasks.workunit.client.0.smithi129.stderr:58284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.f") | .state' 2024-03-25T23:11:24.689 INFO:tasks.workunit.client.0.smithi129.stderr:58284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-25T23:11:24.689 INFO:tasks.workunit.client.0.smithi129.stderr:58284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-25T23:11:24.689 INFO:tasks.workunit.client.0.smithi129.stderr:58284: dumped pgs 2024-03-25T23:11:24.689 INFO:tasks.workunit.client.0.smithi129.stderr:58284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.f 2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.689 INFO:tasks.workunit.client.0.smithi129.stderr:58284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.f 2024-03-25T23:11:24.689 INFO:tasks.workunit.client.0.smithi129.stderr:58284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.689 INFO:tasks.workunit.client.0.smithi129.stderr:58284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-25T23:11:24.690 INFO:tasks.workunit.client.0.smithi129.stderr:58284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-25T23:11:24.690 INFO:tasks.workunit.client.0.smithi129.stderr:58284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-25T23:11:24.690 INFO:tasks.workunit.client.0.smithi129.stderr:58284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-25T23:11:24.690 INFO:tasks.workunit.client.0.smithi129.stderr:58284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-25T23:11:24.690 INFO:tasks.workunit.client.0.smithi129.stderr:58284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.f last_scrub_stamp 2024-03-25T23:11:24.690 INFO:tasks.workunit.client.0.smithi129.stderr:58284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.f 2024-03-25T23:11:24.690 INFO:tasks.workunit.client.0.smithi129.stderr:58284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:24.690 INFO:tasks.workunit.client.0.smithi129.stderr:58284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:24.690 INFO:tasks.workunit.client.0.smithi129.stderr:58284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2024-03-25T23:11:24.690 INFO:tasks.workunit.client.0.smithi129.stderr:58284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-25T23:09:44.508877+0000 '>' '' 2024-03-25T23:11:24.690 INFO:tasks.workunit.client.0.smithi129.stderr:58284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-25T23:11:24.690 INFO:tasks.workunit.client.0.smithi129.stderr:58284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-25T23:11:24.690 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:11:24.708 INFO:tasks.workunit.client.0.smithi129.stderr:58318: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.12 2024-03-25T23:11:24.708 INFO:tasks.workunit.client.0.smithi129.stderr:58318: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.12 2024-03-25T23:11:24.709 INFO:tasks.workunit.client.0.smithi129.stderr:58318: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.12 2024-03-25T23:11:24.709 INFO:tasks.workunit.client.0.smithi129.stderr:58318: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:24.709 INFO:tasks.workunit.client.0.smithi129.stderr:58318: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2024-03-25T23:11:24.709 INFO:tasks.workunit.client.0.smithi129.stderr:58318: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:24.709 INFO:tasks.workunit.client.0.smithi129.stderr:58318: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.709 INFO:tasks.workunit.client.0.smithi129.stderr:58318: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-25T23:11:24.709 INFO:tasks.workunit.client.0.smithi129.stderr:58318: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.12 query 2024-03-25T23:11:24.709 INFO:tasks.workunit.client.0.smithi129.stderr:58318: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-25T23:11:24.709 INFO:tasks.workunit.client.0.smithi129.stderr:58318: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-25T23:11:24.709 INFO:tasks.workunit.client.0.smithi129.stderr:58318: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-25T23:11:24.709 INFO:tasks.workunit.client.0.smithi129.stderr:58318: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.12 2024-03-25T23:11:24.709 INFO:tasks.workunit.client.0.smithi129.stderr:58318: instructing pg 1.12 on osd.0 to scrub 2024-03-25T23:11:24.709 INFO:tasks.workunit.client.0.smithi129.stderr:58318: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-25T23:11:24.710 INFO:tasks.workunit.client.0.smithi129.stderr:58318: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.12") | .state' 2024-03-25T23:11:24.710 INFO:tasks.workunit.client.0.smithi129.stderr:58318: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-25T23:11:24.710 INFO:tasks.workunit.client.0.smithi129.stderr:58318: dumped pgs 2024-03-25T23:11:24.710 INFO:tasks.workunit.client.0.smithi129.stderr:58318: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.12 2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.710 INFO:tasks.workunit.client.0.smithi129.stderr:58318: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.12 2024-03-25T23:11:24.710 INFO:tasks.workunit.client.0.smithi129.stderr:58318: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.710 INFO:tasks.workunit.client.0.smithi129.stderr:58318: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-25T23:11:24.710 INFO:tasks.workunit.client.0.smithi129.stderr:58318: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-25T23:11:24.710 INFO:tasks.workunit.client.0.smithi129.stderr:58318: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-25T23:11:24.710 INFO:tasks.workunit.client.0.smithi129.stderr:58318: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-25T23:11:24.710 INFO:tasks.workunit.client.0.smithi129.stderr:58318: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-25T23:11:24.710 INFO:tasks.workunit.client.0.smithi129.stderr:58318: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.12 last_scrub_stamp 2024-03-25T23:11:24.710 INFO:tasks.workunit.client.0.smithi129.stderr:58318: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.12 2024-03-25T23:11:24.710 INFO:tasks.workunit.client.0.smithi129.stderr:58318: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:24.711 INFO:tasks.workunit.client.0.smithi129.stderr:58318: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:24.711 INFO:tasks.workunit.client.0.smithi129.stderr:58318: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2024-03-25T23:11:24.711 INFO:tasks.workunit.client.0.smithi129.stderr:58318: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-25T23:09:44.508877+0000 '>' '' 2024-03-25T23:11:24.711 INFO:tasks.workunit.client.0.smithi129.stderr:58318: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-25T23:11:24.711 INFO:tasks.workunit.client.0.smithi129.stderr:58318: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-25T23:11:24.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:11:24.717 INFO:tasks.workunit.client.0.smithi129.stderr:58420: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1d 2024-03-25T23:11:24.717 INFO:tasks.workunit.client.0.smithi129.stderr:58420: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1d 2024-03-25T23:11:24.717 INFO:tasks.workunit.client.0.smithi129.stderr:58420: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1d 2024-03-25T23:11:24.717 INFO:tasks.workunit.client.0.smithi129.stderr:58420: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:24.717 INFO:tasks.workunit.client.0.smithi129.stderr:58420: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:24.717 INFO:tasks.workunit.client.0.smithi129.stderr:58420: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2024-03-25T23:11:24.717 INFO:tasks.workunit.client.0.smithi129.stderr:58420: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.717 INFO:tasks.workunit.client.0.smithi129.stderr:58420: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-25T23:11:24.717 INFO:tasks.workunit.client.0.smithi129.stderr:58420: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1d query 2024-03-25T23:11:24.717 INFO:tasks.workunit.client.0.smithi129.stderr:58420: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-25T23:11:24.717 INFO:tasks.workunit.client.0.smithi129.stderr:58420: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-25T23:11:24.718 INFO:tasks.workunit.client.0.smithi129.stderr:58420: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-25T23:11:24.718 INFO:tasks.workunit.client.0.smithi129.stderr:58420: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1d 2024-03-25T23:11:24.718 INFO:tasks.workunit.client.0.smithi129.stderr:58420: instructing pg 1.1d on osd.1 to scrub 2024-03-25T23:11:24.718 INFO:tasks.workunit.client.0.smithi129.stderr:58420: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-25T23:11:24.718 INFO:tasks.workunit.client.0.smithi129.stderr:58420: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-25T23:11:24.718 INFO:tasks.workunit.client.0.smithi129.stderr:58420: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1d") | .state' 2024-03-25T23:11:24.718 INFO:tasks.workunit.client.0.smithi129.stderr:58420: dumped pgs 2024-03-25T23:11:24.718 INFO:tasks.workunit.client.0.smithi129.stderr:58420: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1d 2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.718 INFO:tasks.workunit.client.0.smithi129.stderr:58420: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1d 2024-03-25T23:11:24.718 INFO:tasks.workunit.client.0.smithi129.stderr:58420: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.718 INFO:tasks.workunit.client.0.smithi129.stderr:58420: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-25T23:11:24.718 INFO:tasks.workunit.client.0.smithi129.stderr:58420: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-25T23:11:24.718 INFO:tasks.workunit.client.0.smithi129.stderr:58420: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-25T23:11:24.719 INFO:tasks.workunit.client.0.smithi129.stderr:58420: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-25T23:11:24.719 INFO:tasks.workunit.client.0.smithi129.stderr:58420: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-25T23:11:24.719 INFO:tasks.workunit.client.0.smithi129.stderr:58420: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1d last_scrub_stamp 2024-03-25T23:11:24.719 INFO:tasks.workunit.client.0.smithi129.stderr:58420: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1d 2024-03-25T23:11:24.719 INFO:tasks.workunit.client.0.smithi129.stderr:58420: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:24.719 INFO:tasks.workunit.client.0.smithi129.stderr:58420: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2024-03-25T23:11:24.719 INFO:tasks.workunit.client.0.smithi129.stderr:58420: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:24.719 INFO:tasks.workunit.client.0.smithi129.stderr:58420: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-25T23:09:44.508877+0000 '>' '' 2024-03-25T23:11:24.719 INFO:tasks.workunit.client.0.smithi129.stderr:58420: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-25T23:11:24.719 INFO:tasks.workunit.client.0.smithi129.stderr:58420: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-25T23:11:24.719 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:11:24.768 INFO:tasks.workunit.client.0.smithi129.stderr:58323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.14 2024-03-25T23:11:24.768 INFO:tasks.workunit.client.0.smithi129.stderr:58323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.14 2024-03-25T23:11:24.768 INFO:tasks.workunit.client.0.smithi129.stderr:58323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.14 2024-03-25T23:11:24.768 INFO:tasks.workunit.client.0.smithi129.stderr:58323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:24.769 INFO:tasks.workunit.client.0.smithi129.stderr:58323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2024-03-25T23:11:24.769 INFO:tasks.workunit.client.0.smithi129.stderr:58323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:24.769 INFO:tasks.workunit.client.0.smithi129.stderr:58323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.769 INFO:tasks.workunit.client.0.smithi129.stderr:58323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-25T23:11:24.769 INFO:tasks.workunit.client.0.smithi129.stderr:58323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.14 query 2024-03-25T23:11:24.769 INFO:tasks.workunit.client.0.smithi129.stderr:58323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-25T23:11:24.769 INFO:tasks.workunit.client.0.smithi129.stderr:58323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-25T23:11:24.769 INFO:tasks.workunit.client.0.smithi129.stderr:58323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-25T23:11:24.769 INFO:tasks.workunit.client.0.smithi129.stderr:58323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.14 2024-03-25T23:11:24.769 INFO:tasks.workunit.client.0.smithi129.stderr:58323: instructing pg 1.14 on osd.2 to scrub 2024-03-25T23:11:24.769 INFO:tasks.workunit.client.0.smithi129.stderr:58323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.14") | .state' 2024-03-25T23:11:24.769 INFO:tasks.workunit.client.0.smithi129.stderr:58323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-25T23:11:24.769 INFO:tasks.workunit.client.0.smithi129.stderr:58323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-25T23:11:24.770 INFO:tasks.workunit.client.0.smithi129.stderr:58323: dumped pgs 2024-03-25T23:11:24.770 INFO:tasks.workunit.client.0.smithi129.stderr:58323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.14 2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.770 INFO:tasks.workunit.client.0.smithi129.stderr:58323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.14 2024-03-25T23:11:24.770 INFO:tasks.workunit.client.0.smithi129.stderr:58323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.770 INFO:tasks.workunit.client.0.smithi129.stderr:58323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-25T23:11:24.770 INFO:tasks.workunit.client.0.smithi129.stderr:58323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-25T23:11:24.770 INFO:tasks.workunit.client.0.smithi129.stderr:58323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-25T23:11:24.770 INFO:tasks.workunit.client.0.smithi129.stderr:58323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-25T23:11:24.770 INFO:tasks.workunit.client.0.smithi129.stderr:58323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-25T23:11:24.770 INFO:tasks.workunit.client.0.smithi129.stderr:58323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.14 last_scrub_stamp 2024-03-25T23:11:24.770 INFO:tasks.workunit.client.0.smithi129.stderr:58323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.14 2024-03-25T23:11:24.770 INFO:tasks.workunit.client.0.smithi129.stderr:58323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:24.770 INFO:tasks.workunit.client.0.smithi129.stderr:58323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2024-03-25T23:11:24.770 INFO:tasks.workunit.client.0.smithi129.stderr:58323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:24.771 INFO:tasks.workunit.client.0.smithi129.stderr:58323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-25T23:09:44.508877+0000 '>' '' 2024-03-25T23:11:24.771 INFO:tasks.workunit.client.0.smithi129.stderr:58323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-25T23:11:24.771 INFO:tasks.workunit.client.0.smithi129.stderr:58323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-25T23:11:24.771 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:11:24.774 INFO:tasks.workunit.client.0.smithi129.stderr:58346: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.17 2024-03-25T23:11:24.774 INFO:tasks.workunit.client.0.smithi129.stderr:58346: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.17 2024-03-25T23:11:24.774 INFO:tasks.workunit.client.0.smithi129.stderr:58346: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.17 2024-03-25T23:11:24.774 INFO:tasks.workunit.client.0.smithi129.stderr:58346: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:24.775 INFO:tasks.workunit.client.0.smithi129.stderr:58346: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2024-03-25T23:11:24.775 INFO:tasks.workunit.client.0.smithi129.stderr:58346: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:24.775 INFO:tasks.workunit.client.0.smithi129.stderr:58346: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.775 INFO:tasks.workunit.client.0.smithi129.stderr:58346: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.17 query 2024-03-25T23:11:24.775 INFO:tasks.workunit.client.0.smithi129.stderr:58346: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-25T23:11:24.775 INFO:tasks.workunit.client.0.smithi129.stderr:58346: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-25T23:11:24.775 INFO:tasks.workunit.client.0.smithi129.stderr:58346: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-25T23:11:24.775 INFO:tasks.workunit.client.0.smithi129.stderr:58346: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-25T23:11:24.775 INFO:tasks.workunit.client.0.smithi129.stderr:58346: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.17 2024-03-25T23:11:24.775 INFO:tasks.workunit.client.0.smithi129.stderr:58346: instructing pg 1.17 on osd.0 to scrub 2024-03-25T23:11:24.775 INFO:tasks.workunit.client.0.smithi129.stderr:58346: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-25T23:11:24.775 INFO:tasks.workunit.client.0.smithi129.stderr:58346: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-25T23:11:24.775 INFO:tasks.workunit.client.0.smithi129.stderr:58346: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.17") | .state' 2024-03-25T23:11:24.775 INFO:tasks.workunit.client.0.smithi129.stderr:58346: dumped pgs 2024-03-25T23:11:24.776 INFO:tasks.workunit.client.0.smithi129.stderr:58346: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.17 2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.776 INFO:tasks.workunit.client.0.smithi129.stderr:58346: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.17 2024-03-25T23:11:24.776 INFO:tasks.workunit.client.0.smithi129.stderr:58346: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.776 INFO:tasks.workunit.client.0.smithi129.stderr:58346: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-25T23:11:24.776 INFO:tasks.workunit.client.0.smithi129.stderr:58346: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-25T23:11:24.776 INFO:tasks.workunit.client.0.smithi129.stderr:58346: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-25T23:11:24.776 INFO:tasks.workunit.client.0.smithi129.stderr:58346: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-25T23:11:24.776 INFO:tasks.workunit.client.0.smithi129.stderr:58346: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-25T23:11:24.776 INFO:tasks.workunit.client.0.smithi129.stderr:58346: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.17 last_scrub_stamp 2024-03-25T23:11:24.776 INFO:tasks.workunit.client.0.smithi129.stderr:58346: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.17 2024-03-25T23:11:24.776 INFO:tasks.workunit.client.0.smithi129.stderr:58346: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:24.776 INFO:tasks.workunit.client.0.smithi129.stderr:58346: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:24.776 INFO:tasks.workunit.client.0.smithi129.stderr:58346: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2024-03-25T23:11:24.776 INFO:tasks.workunit.client.0.smithi129.stderr:58346: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-25T23:09:44.508877+0000 '>' '' 2024-03-25T23:11:24.777 INFO:tasks.workunit.client.0.smithi129.stderr:58346: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-25T23:11:24.777 INFO:tasks.workunit.client.0.smithi129.stderr:58346: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-25T23:11:24.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:11:24.777 INFO:tasks.workunit.client.0.smithi129.stderr:58326: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.15 2024-03-25T23:11:24.777 INFO:tasks.workunit.client.0.smithi129.stderr:58326: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.15 2024-03-25T23:11:24.777 INFO:tasks.workunit.client.0.smithi129.stderr:58326: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.15 2024-03-25T23:11:24.777 INFO:tasks.workunit.client.0.smithi129.stderr:58326: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:24.778 INFO:tasks.workunit.client.0.smithi129.stderr:58326: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:24.778 INFO:tasks.workunit.client.0.smithi129.stderr:58326: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2024-03-25T23:11:24.778 INFO:tasks.workunit.client.0.smithi129.stderr:58326: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.778 INFO:tasks.workunit.client.0.smithi129.stderr:58326: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.15 query 2024-03-25T23:11:24.778 INFO:tasks.workunit.client.0.smithi129.stderr:58326: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-25T23:11:24.778 INFO:tasks.workunit.client.0.smithi129.stderr:58326: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-25T23:11:24.778 INFO:tasks.workunit.client.0.smithi129.stderr:58326: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-25T23:11:24.778 INFO:tasks.workunit.client.0.smithi129.stderr:58326: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-25T23:11:24.778 INFO:tasks.workunit.client.0.smithi129.stderr:58326: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.15 2024-03-25T23:11:24.778 INFO:tasks.workunit.client.0.smithi129.stderr:58326: instructing pg 1.15 on osd.1 to scrub 2024-03-25T23:11:24.778 INFO:tasks.workunit.client.0.smithi129.stderr:58326: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-25T23:11:24.778 INFO:tasks.workunit.client.0.smithi129.stderr:58326: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.15") | .state' 2024-03-25T23:11:24.778 INFO:tasks.workunit.client.0.smithi129.stderr:58326: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-25T23:11:24.779 INFO:tasks.workunit.client.0.smithi129.stderr:58326: dumped pgs 2024-03-25T23:11:24.779 INFO:tasks.workunit.client.0.smithi129.stderr:58326: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.15 2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.779 INFO:tasks.workunit.client.0.smithi129.stderr:58326: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.15 2024-03-25T23:11:24.779 INFO:tasks.workunit.client.0.smithi129.stderr:58326: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:24.779 INFO:tasks.workunit.client.0.smithi129.stderr:58326: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-25T23:11:24.779 INFO:tasks.workunit.client.0.smithi129.stderr:58326: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-25T23:11:24.779 INFO:tasks.workunit.client.0.smithi129.stderr:58326: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-25T23:11:24.780 INFO:tasks.workunit.client.0.smithi129.stderr:58326: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-25T23:11:24.780 INFO:tasks.workunit.client.0.smithi129.stderr:58326: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-25T23:11:24.780 INFO:tasks.workunit.client.0.smithi129.stderr:58326: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.15 last_scrub_stamp 2024-03-25T23:11:24.780 INFO:tasks.workunit.client.0.smithi129.stderr:58326: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.15 2024-03-25T23:11:24.780 INFO:tasks.workunit.client.0.smithi129.stderr:58326: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:24.780 INFO:tasks.workunit.client.0.smithi129.stderr:58326: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2024-03-25T23:11:24.780 INFO:tasks.workunit.client.0.smithi129.stderr:58326: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:24.780 INFO:tasks.workunit.client.0.smithi129.stderr:58326: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-25T23:09:44.508877+0000 '>' '' 2024-03-25T23:11:24.780 INFO:tasks.workunit.client.0.smithi129.stderr:58326: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-25T23:11:24.780 INFO:tasks.workunit.client.0.smithi129.stderr:58326: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-25T23:11:24.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:11:25.161 INFO:tasks.workunit.client.0.smithi129.stderr:et_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:25.162 INFO:tasks.workunit.client.0.smithi129.stderr:58430: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-25T23:09:44.508877+0000 '>' '' 2024-03-25T23:11:25.162 INFO:tasks.workunit.client.0.smithi129.stderr:58430: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-25T23:11:25.162 INFO:tasks.workunit.client.0.smithi129.stderr:58430: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test true = true 2024-03-25T23:11:25.162 INFO:tasks.workunit.client.0.smithi129.stderr:58430: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:203: pg_scrub_mod: return 2 2024-03-25T23:11:25.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 2 2024-03-25T23:11:25.186 INFO:tasks.workunit.client.0.smithi129.stderr:58297: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.10 2024-03-25T23:11:25.186 INFO:tasks.workunit.client.0.smithi129.stderr:58297: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.10 2024-03-25T23:11:25.186 INFO:tasks.workunit.client.0.smithi129.stderr:58297: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.10 2024-03-25T23:11:25.186 INFO:tasks.workunit.client.0.smithi129.stderr:58297: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:25.186 INFO:tasks.workunit.client.0.smithi129.stderr:58297: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:25.187 INFO:tasks.workunit.client.0.smithi129.stderr:58297: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2024-03-25T23:11:25.187 INFO:tasks.workunit.client.0.smithi129.stderr:58297: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:25.187 INFO:tasks.workunit.client.0.smithi129.stderr:58297: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.10 query 2024-03-25T23:11:25.187 INFO:tasks.workunit.client.0.smithi129.stderr:58297: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-25T23:11:25.187 INFO:tasks.workunit.client.0.smithi129.stderr:58297: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-25T23:11:25.187 INFO:tasks.workunit.client.0.smithi129.stderr:58297: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-25T23:11:25.187 INFO:tasks.workunit.client.0.smithi129.stderr:58297: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-25T23:11:25.187 INFO:tasks.workunit.client.0.smithi129.stderr:58297: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.10 2024-03-25T23:11:25.187 INFO:tasks.workunit.client.0.smithi129.stderr:58297: instructing pg 1.10 on osd.5 to scrub 2024-03-25T23:11:25.187 INFO:tasks.workunit.client.0.smithi129.stderr:58297: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.10") | .state' 2024-03-25T23:11:25.187 INFO:tasks.workunit.client.0.smithi129.stderr:58297: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-25T23:11:25.187 INFO:tasks.workunit.client.0.smithi129.stderr:58297: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-25T23:11:25.187 INFO:tasks.workunit.client.0.smithi129.stderr:58297: dumped pgs 2024-03-25T23:11:25.187 INFO:tasks.workunit.client.0.smithi129.stderr:58297: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.10 2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:25.188 INFO:tasks.workunit.client.0.smithi129.stderr:58297: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.10 2024-03-25T23:11:25.188 INFO:tasks.workunit.client.0.smithi129.stderr:58297: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:25.188 INFO:tasks.workunit.client.0.smithi129.stderr:58297: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-25T23:11:25.188 INFO:tasks.workunit.client.0.smithi129.stderr:58297: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-25T23:11:25.188 INFO:tasks.workunit.client.0.smithi129.stderr:58297: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-25T23:11:25.188 INFO:tasks.workunit.client.0.smithi129.stderr:58297: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-25T23:11:25.188 INFO:tasks.workunit.client.0.smithi129.stderr:58297: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-25T23:11:25.188 INFO:tasks.workunit.client.0.smithi129.stderr:58297: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.10 last_scrub_stamp 2024-03-25T23:11:25.188 INFO:tasks.workunit.client.0.smithi129.stderr:58297: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.10 2024-03-25T23:11:25.188 INFO:tasks.workunit.client.0.smithi129.stderr:58297: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:25.188 INFO:tasks.workunit.client.0.smithi129.stderr:58297: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2024-03-25T23:11:25.188 INFO:tasks.workunit.client.0.smithi129.stderr:58297: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:25.188 INFO:tasks.workunit.client.0.smithi129.stderr:58297: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-25T23:09:44.508877+0000 '>' '' 2024-03-25T23:11:25.188 INFO:tasks.workunit.client.0.smithi129.stderr:58297: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-25T23:11:25.188 INFO:tasks.workunit.client.0.smithi129.stderr:58297: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-25T23:11:25.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:11:25.212 INFO:tasks.workunit.client.0.smithi129.stderr:58300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.11 2024-03-25T23:11:25.212 INFO:tasks.workunit.client.0.smithi129.stderr:58300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.11 2024-03-25T23:11:25.212 INFO:tasks.workunit.client.0.smithi129.stderr:58300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.11 2024-03-25T23:11:25.212 INFO:tasks.workunit.client.0.smithi129.stderr:58300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:25.212 INFO:tasks.workunit.client.0.smithi129.stderr:58300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:25.212 INFO:tasks.workunit.client.0.smithi129.stderr:58300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2024-03-25T23:11:25.212 INFO:tasks.workunit.client.0.smithi129.stderr:58300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:25.213 INFO:tasks.workunit.client.0.smithi129.stderr:58300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.11 query 2024-03-25T23:11:25.213 INFO:tasks.workunit.client.0.smithi129.stderr:58300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-25T23:11:25.213 INFO:tasks.workunit.client.0.smithi129.stderr:58300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-25T23:11:25.213 INFO:tasks.workunit.client.0.smithi129.stderr:58300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-25T23:11:25.213 INFO:tasks.workunit.client.0.smithi129.stderr:58300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-25T23:11:25.213 INFO:tasks.workunit.client.0.smithi129.stderr:58300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.11 2024-03-25T23:11:25.213 INFO:tasks.workunit.client.0.smithi129.stderr:58300: instructing pg 1.11 on osd.0 to scrub 2024-03-25T23:11:25.213 INFO:tasks.workunit.client.0.smithi129.stderr:58300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.11") | .state' 2024-03-25T23:11:25.213 INFO:tasks.workunit.client.0.smithi129.stderr:58300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-25T23:11:25.213 INFO:tasks.workunit.client.0.smithi129.stderr:58300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-25T23:11:25.213 INFO:tasks.workunit.client.0.smithi129.stderr:58300: dumped pgs 2024-03-25T23:11:25.213 INFO:tasks.workunit.client.0.smithi129.stderr:58300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.11 2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:25.213 INFO:tasks.workunit.client.0.smithi129.stderr:58300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.11 2024-03-25T23:11:25.213 INFO:tasks.workunit.client.0.smithi129.stderr:58300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:25.214 INFO:tasks.workunit.client.0.smithi129.stderr:58300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-25T23:11:25.214 INFO:tasks.workunit.client.0.smithi129.stderr:58300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-25T23:11:25.214 INFO:tasks.workunit.client.0.smithi129.stderr:58300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-25T23:11:25.214 INFO:tasks.workunit.client.0.smithi129.stderr:58300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-25T23:11:25.214 INFO:tasks.workunit.client.0.smithi129.stderr:58300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-25T23:11:25.214 INFO:tasks.workunit.client.0.smithi129.stderr:58300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.11 last_scrub_stamp 2024-03-25T23:11:25.214 INFO:tasks.workunit.client.0.smithi129.stderr:58300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.11 2024-03-25T23:11:25.214 INFO:tasks.workunit.client.0.smithi129.stderr:58300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:25.214 INFO:tasks.workunit.client.0.smithi129.stderr:58300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2024-03-25T23:11:25.214 INFO:tasks.workunit.client.0.smithi129.stderr:58300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:25.214 INFO:tasks.workunit.client.0.smithi129.stderr:58300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-25T23:09:44.508877+0000 '>' '' 2024-03-25T23:11:25.214 INFO:tasks.workunit.client.0.smithi129.stderr:58300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-25T23:11:25.214 INFO:tasks.workunit.client.0.smithi129.stderr:58300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-25T23:11:25.214 INFO:tasks.workunit.client.0.smithi129.stderr:58440: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1f 2024-03-25T23:11:25.215 INFO:tasks.workunit.client.0.smithi129.stderr:58440: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1f 2024-03-25T23:11:25.215 INFO:tasks.workunit.client.0.smithi129.stderr:58440: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1f 2024-03-25T23:11:25.215 INFO:tasks.workunit.client.0.smithi129.stderr:58440: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:25.215 INFO:tasks.workunit.client.0.smithi129.stderr:58440: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:25.215 INFO:tasks.workunit.client.0.smithi129.stderr:58440: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2024-03-25T23:11:25.215 INFO:tasks.workunit.client.0.smithi129.stderr:58440: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:25.215 INFO:tasks.workunit.client.0.smithi129.stderr:58440: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1f query 2024-03-25T23:11:25.215 INFO:tasks.workunit.client.0.smithi129.stderr:58440: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-25T23:11:25.215 INFO:tasks.workunit.client.0.smithi129.stderr:58440: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-25T23:11:25.215 INFO:tasks.workunit.client.0.smithi129.stderr:58440: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-25T23:11:25.215 INFO:tasks.workunit.client.0.smithi129.stderr:58440: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-25T23:11:25.215 INFO:tasks.workunit.client.0.smithi129.stderr:58440: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1f 2024-03-25T23:11:25.215 INFO:tasks.workunit.client.0.smithi129.stderr:58440: instructing pg 1.1f on osd.5 to scrub 2024-03-25T23:11:25.215 INFO:tasks.workunit.client.0.smithi129.stderr:58440: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-25T23:11:25.216 INFO:tasks.workunit.client.0.smithi129.stderr:58440: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1f") | .state' 2024-03-25T23:11:25.216 INFO:tasks.workunit.client.0.smithi129.stderr:58440: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-25T23:11:25.216 INFO:tasks.workunit.client.0.smithi129.stderr:58440: dumped pgs 2024-03-25T23:11:25.216 INFO:tasks.workunit.client.0.smithi129.stderr:58440: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1f 2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:25.216 INFO:tasks.workunit.client.0.smithi129.stderr:58440: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1f 2024-03-25T23:11:25.216 INFO:tasks.workunit.client.0.smithi129.stderr:58440: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:25.216 INFO:tasks.workunit.client.0.smithi129.stderr:58440: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-25T23:11:25.216 INFO:tasks.workunit.client.0.smithi129.stderr:58440: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-25T23:11:25.216 INFO:tasks.workunit.client.0.smithi129.stderr:58440: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-25T23:11:25.216 INFO:tasks.workunit.client.0.smithi129.stderr:58440: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-25T23:11:25.216 INFO:tasks.workunit.client.0.smithi129.stderr:58440: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-25T23:11:25.216 INFO:tasks.workunit.client.0.smithi129.stderr:58440: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1f last_scrub_stamp 2024-03-25T23:11:25.216 INFO:tasks.workunit.client.0.smithi129.stderr:58440: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1f 2024-03-25T23:11:25.216 INFO:tasks.workunit.client.0.smithi129.stderr:58440: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:25.216 INFO:tasks.workunit.client.0.smithi129.stderr:58440: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:25.217 INFO:tasks.workunit.client.0.smithi129.stderr:58440: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2024-03-25T23:11:25.217 INFO:tasks.workunit.client.0.smithi129.stderr:58440: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-25T23:09:44.508877+0000 '>' '' 2024-03-25T23:11:25.217 INFO:tasks.workunit.client.0.smithi129.stderr:58440: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-25T23:11:25.217 INFO:tasks.workunit.client.0.smithi129.stderr:58440: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-25T23:11:25.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:11:25.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:11:25.220 INFO:tasks.workunit.client.0.smithi129.stderr:58374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1a 2024-03-25T23:11:25.220 INFO:tasks.workunit.client.0.smithi129.stderr:58374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1a 2024-03-25T23:11:25.220 INFO:tasks.workunit.client.0.smithi129.stderr:58374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1a 2024-03-25T23:11:25.220 INFO:tasks.workunit.client.0.smithi129.stderr:58374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:25.220 INFO:tasks.workunit.client.0.smithi129.stderr:58374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2024-03-25T23:11:25.220 INFO:tasks.workunit.client.0.smithi129.stderr:58374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:25.220 INFO:tasks.workunit.client.0.smithi129.stderr:58374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:25.220 INFO:tasks.workunit.client.0.smithi129.stderr:58374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-25T23:11:25.220 INFO:tasks.workunit.client.0.smithi129.stderr:58374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1a query 2024-03-25T23:11:25.221 INFO:tasks.workunit.client.0.smithi129.stderr:58374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-25T23:11:25.221 INFO:tasks.workunit.client.0.smithi129.stderr:58374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-25T23:11:25.221 INFO:tasks.workunit.client.0.smithi129.stderr:58374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-25T23:11:25.221 INFO:tasks.workunit.client.0.smithi129.stderr:58374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1a 2024-03-25T23:11:25.221 INFO:tasks.workunit.client.0.smithi129.stderr:58374: instructing pg 1.1a on osd.1 to scrub 2024-03-25T23:11:25.221 INFO:tasks.workunit.client.0.smithi129.stderr:58374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-25T23:11:25.221 INFO:tasks.workunit.client.0.smithi129.stderr:58374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1a") | .state' 2024-03-25T23:11:25.221 INFO:tasks.workunit.client.0.smithi129.stderr:58374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-25T23:11:25.221 INFO:tasks.workunit.client.0.smithi129.stderr:58374: dumped pgs 2024-03-25T23:11:25.221 INFO:tasks.workunit.client.0.smithi129.stderr:58374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1a 2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:25.221 INFO:tasks.workunit.client.0.smithi129.stderr:58374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1a 2024-03-25T23:11:25.221 INFO:tasks.workunit.client.0.smithi129.stderr:58374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:25.221 INFO:tasks.workunit.client.0.smithi129.stderr:58374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-25T23:11:25.221 INFO:tasks.workunit.client.0.smithi129.stderr:58374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-25T23:11:25.222 INFO:tasks.workunit.client.0.smithi129.stderr:58374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-25T23:11:25.222 INFO:tasks.workunit.client.0.smithi129.stderr:58374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-25T23:11:25.222 INFO:tasks.workunit.client.0.smithi129.stderr:58374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-25T23:11:25.222 INFO:tasks.workunit.client.0.smithi129.stderr:58374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1a last_scrub_stamp 2024-03-25T23:11:25.222 INFO:tasks.workunit.client.0.smithi129.stderr:58374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1a 2024-03-25T23:11:25.222 INFO:tasks.workunit.client.0.smithi129.stderr:58374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:25.222 INFO:tasks.workunit.client.0.smithi129.stderr:58374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:25.222 INFO:tasks.workunit.client.0.smithi129.stderr:58374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2024-03-25T23:11:25.222 INFO:tasks.workunit.client.0.smithi129.stderr:58374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-25T23:11:21.668521+0000 '>' '' 2024-03-25T23:11:25.222 INFO:tasks.workunit.client.0.smithi129.stderr:58374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-25T23:11:25.222 INFO:tasks.workunit.client.0.smithi129.stderr:58374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-25T23:11:25.222 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:11:25.223 INFO:tasks.workunit.client.0.smithi129.stderr:58245: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.8 2024-03-25T23:11:25.223 INFO:tasks.workunit.client.0.smithi129.stderr:58245: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.8 2024-03-25T23:11:25.223 INFO:tasks.workunit.client.0.smithi129.stderr:58245: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.8 2024-03-25T23:11:25.223 INFO:tasks.workunit.client.0.smithi129.stderr:58245: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:25.224 INFO:tasks.workunit.client.0.smithi129.stderr:58245: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:25.224 INFO:tasks.workunit.client.0.smithi129.stderr:58245: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2024-03-25T23:11:25.224 INFO:tasks.workunit.client.0.smithi129.stderr:58245: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:25.224 INFO:tasks.workunit.client.0.smithi129.stderr:58245: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.8 query 2024-03-25T23:11:25.224 INFO:tasks.workunit.client.0.smithi129.stderr:58245: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-25T23:11:25.224 INFO:tasks.workunit.client.0.smithi129.stderr:58245: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-25T23:11:25.224 INFO:tasks.workunit.client.0.smithi129.stderr:58245: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-25T23:11:25.224 INFO:tasks.workunit.client.0.smithi129.stderr:58245: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-25T23:11:25.224 INFO:tasks.workunit.client.0.smithi129.stderr:58245: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.8 2024-03-25T23:11:25.224 INFO:tasks.workunit.client.0.smithi129.stderr:58245: instructing pg 1.8 on osd.5 to scrub 2024-03-25T23:11:25.224 INFO:tasks.workunit.client.0.smithi129.stderr:58245: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.8") | .state' 2024-03-25T23:11:25.224 INFO:tasks.workunit.client.0.smithi129.stderr:58245: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-25T23:11:25.224 INFO:tasks.workunit.client.0.smithi129.stderr:58245: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-25T23:11:25.224 INFO:tasks.workunit.client.0.smithi129.stderr:58245: dumped pgs 2024-03-25T23:11:25.225 INFO:tasks.workunit.client.0.smithi129.stderr:58245: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.8 2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:25.225 INFO:tasks.workunit.client.0.smithi129.stderr:58245: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.8 2024-03-25T23:11:25.225 INFO:tasks.workunit.client.0.smithi129.stderr:58245: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:25.225 INFO:tasks.workunit.client.0.smithi129.stderr:58245: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-25T23:11:25.225 INFO:tasks.workunit.client.0.smithi129.stderr:58245: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-25T23:11:25.225 INFO:tasks.workunit.client.0.smithi129.stderr:58245: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-25T23:11:25.225 INFO:tasks.workunit.client.0.smithi129.stderr:58245: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-25T23:11:25.225 INFO:tasks.workunit.client.0.smithi129.stderr:58245: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-25T23:11:25.225 INFO:tasks.workunit.client.0.smithi129.stderr:58245: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.8 last_scrub_stamp 2024-03-25T23:11:25.225 INFO:tasks.workunit.client.0.smithi129.stderr:58245: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.8 2024-03-25T23:11:25.225 INFO:tasks.workunit.client.0.smithi129.stderr:58245: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:25.225 INFO:tasks.workunit.client.0.smithi129.stderr:58245: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2024-03-25T23:11:25.225 INFO:tasks.workunit.client.0.smithi129.stderr:58245: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:25.225 INFO:tasks.workunit.client.0.smithi129.stderr:58245: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-25T23:09:44.508877+0000 '>' '' 2024-03-25T23:11:25.225 INFO:tasks.workunit.client.0.smithi129.stderr:58245: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-25T23:11:25.226 INFO:tasks.workunit.client.0.smithi129.stderr:58245: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-25T23:11:25.226 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:11:25.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-25T23:11:25.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-25T23:11:25.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-25T23:11:25.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-25T23:11:25.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58234 2024-03-25T23:11:25.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-25T23:11:25.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-25T23:11:25.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-25T23:11:25.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-25T23:11:25.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58241 2024-03-25T23:11:25.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-25T23:11:25.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-25T23:11:25.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-25T23:11:25.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-25T23:11:25.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58247 2024-03-25T23:11:25.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-25T23:11:25.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-25T23:11:25.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-25T23:11:25.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-25T23:11:25.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58262 2024-03-25T23:11:25.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-25T23:11:25.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-25T23:11:25.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-25T23:11:25.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-25T23:11:25.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58271 2024-03-25T23:11:25.252 INFO:tasks.workunit.client.0.smithi129.stderr:58386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1b 2024-03-25T23:11:25.252 INFO:tasks.workunit.client.0.smithi129.stderr:58386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1b 2024-03-25T23:11:25.252 INFO:tasks.workunit.client.0.smithi129.stderr:58386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1b 2024-03-25T23:11:25.252 INFO:tasks.workunit.client.0.smithi129.stderr:58386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:25.252 INFO:tasks.workunit.client.0.smithi129.stderr:58386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2024-03-25T23:11:25.252 INFO:tasks.workunit.client.0.smithi129.stderr:58386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:25.252 INFO:tasks.workunit.client.0.smithi129.stderr:58386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:25.252 INFO:tasks.workunit.client.0.smithi129.stderr:58386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1b query 2024-03-25T23:11:25.252 INFO:tasks.workunit.client.0.smithi129.stderr:58386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-25T23:11:25.252 INFO:tasks.workunit.client.0.smithi129.stderr:58386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-25T23:11:25.252 INFO:tasks.workunit.client.0.smithi129.stderr:58386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-25T23:11:25.253 INFO:tasks.workunit.client.0.smithi129.stderr:58386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-25T23:11:25.253 INFO:tasks.workunit.client.0.smithi129.stderr:58386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1b 2024-03-25T23:11:25.253 INFO:tasks.workunit.client.0.smithi129.stderr:58386: instructing pg 1.1b on osd.0 to scrub 2024-03-25T23:11:25.253 INFO:tasks.workunit.client.0.smithi129.stderr:58386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-25T23:11:25.253 INFO:tasks.workunit.client.0.smithi129.stderr:58386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1b") | .state' 2024-03-25T23:11:25.253 INFO:tasks.workunit.client.0.smithi129.stderr:58386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-25T23:11:25.253 INFO:tasks.workunit.client.0.smithi129.stderr:58386: dumped pgs 2024-03-25T23:11:25.253 INFO:tasks.workunit.client.0.smithi129.stderr:58386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1b 2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:25.253 INFO:tasks.workunit.client.0.smithi129.stderr:58386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1b 2024-03-25T23:11:25.253 INFO:tasks.workunit.client.0.smithi129.stderr:58386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:25.253 INFO:tasks.workunit.client.0.smithi129.stderr:58386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-25T23:11:25.253 INFO:tasks.workunit.client.0.smithi129.stderr:58386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-25T23:11:25.253 INFO:tasks.workunit.client.0.smithi129.stderr:58386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-25T23:11:25.253 INFO:tasks.workunit.client.0.smithi129.stderr:58386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-25T23:11:25.253 INFO:tasks.workunit.client.0.smithi129.stderr:58386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-25T23:11:25.254 INFO:tasks.workunit.client.0.smithi129.stderr:58386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1b last_scrub_stamp 2024-03-25T23:11:25.254 INFO:tasks.workunit.client.0.smithi129.stderr:58386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1b 2024-03-25T23:11:25.254 INFO:tasks.workunit.client.0.smithi129.stderr:58386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:25.254 INFO:tasks.workunit.client.0.smithi129.stderr:58386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2024-03-25T23:11:25.254 INFO:tasks.workunit.client.0.smithi129.stderr:58386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:25.254 INFO:tasks.workunit.client.0.smithi129.stderr:58386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-25T23:09:44.508877+0000 '>' '' 2024-03-25T23:11:25.254 INFO:tasks.workunit.client.0.smithi129.stderr:58386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-25T23:11:25.254 INFO:tasks.workunit.client.0.smithi129.stderr:58386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-25T23:11:25.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:11:25.278 INFO:tasks.workunit.client.0.smithi129.stderr:lect(.pgid=="1.d") | .last_scrub_stamp' 2024-03-25T23:11:25.278 INFO:tasks.workunit.client.0.smithi129.stderr:58278: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-25T23:09:44.508877+0000 '>' '' 2024-03-25T23:11:25.278 INFO:tasks.workunit.client.0.smithi129.stderr:58278: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-25T23:11:25.279 INFO:tasks.workunit.client.0.smithi129.stderr:58278: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test true = true 2024-03-25T23:11:25.279 INFO:tasks.workunit.client.0.smithi129.stderr:58278: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:203: pg_scrub_mod: return 2 2024-03-25T23:11:25.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 2 2024-03-25T23:11:25.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=2 2024-03-25T23:11:25.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 2 -eq 2 2024-03-25T23:11:25.279 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: expr 0 + 1 2024-03-25T23:11:25.282 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: recov_scrub_count=1 2024-03-25T23:11:25.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-25T23:11:25.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58275 2024-03-25T23:11:25.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-25T23:11:25.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-25T23:11:25.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-25T23:11:25.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-25T23:11:25.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58281 2024-03-25T23:11:25.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-25T23:11:25.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-25T23:11:25.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-25T23:11:25.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-25T23:11:25.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58291 2024-03-25T23:11:25.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-25T23:11:25.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-25T23:11:25.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-25T23:11:25.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-25T23:11:25.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58294 2024-03-25T23:11:25.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-25T23:11:25.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-25T23:11:25.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-25T23:11:25.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-25T23:11:25.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58301 2024-03-25T23:11:25.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-25T23:11:25.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-25T23:11:25.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-25T23:11:25.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-25T23:11:25.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58310 2024-03-25T23:11:25.298 INFO:tasks.workunit.client.0.smithi129.stderr:58362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.19 2024-03-25T23:11:25.298 INFO:tasks.workunit.client.0.smithi129.stderr:58362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.19 2024-03-25T23:11:25.298 INFO:tasks.workunit.client.0.smithi129.stderr:58362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.19 2024-03-25T23:11:25.298 INFO:tasks.workunit.client.0.smithi129.stderr:58362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:25.298 INFO:tasks.workunit.client.0.smithi129.stderr:58362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2024-03-25T23:11:25.299 INFO:tasks.workunit.client.0.smithi129.stderr:58362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:25.299 INFO:tasks.workunit.client.0.smithi129.stderr:58362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:25.299 INFO:tasks.workunit.client.0.smithi129.stderr:58362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.19 query 2024-03-25T23:11:25.299 INFO:tasks.workunit.client.0.smithi129.stderr:58362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-25T23:11:25.299 INFO:tasks.workunit.client.0.smithi129.stderr:58362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-25T23:11:25.299 INFO:tasks.workunit.client.0.smithi129.stderr:58362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-25T23:11:25.299 INFO:tasks.workunit.client.0.smithi129.stderr:58362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-25T23:11:25.299 INFO:tasks.workunit.client.0.smithi129.stderr:58362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.19 2024-03-25T23:11:25.299 INFO:tasks.workunit.client.0.smithi129.stderr:58362: instructing pg 1.19 on osd.3 to scrub 2024-03-25T23:11:25.299 INFO:tasks.workunit.client.0.smithi129.stderr:58362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.19") | .state' 2024-03-25T23:11:25.299 INFO:tasks.workunit.client.0.smithi129.stderr:58362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-25T23:11:25.299 INFO:tasks.workunit.client.0.smithi129.stderr:58362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-25T23:11:25.299 INFO:tasks.workunit.client.0.smithi129.stderr:58362: dumped pgs 2024-03-25T23:11:25.300 INFO:tasks.workunit.client.0.smithi129.stderr:58362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.19 2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:25.300 INFO:tasks.workunit.client.0.smithi129.stderr:58362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.19 2024-03-25T23:11:25.300 INFO:tasks.workunit.client.0.smithi129.stderr:58362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:25.300 INFO:tasks.workunit.client.0.smithi129.stderr:58362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-25T23:11:25.301 INFO:tasks.workunit.client.0.smithi129.stderr:58362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-25T23:11:25.301 INFO:tasks.workunit.client.0.smithi129.stderr:58362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-25T23:11:25.302 INFO:tasks.workunit.client.0.smithi129.stderr:58362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-25T23:11:25.302 INFO:tasks.workunit.client.0.smithi129.stderr:58362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-25T23:11:25.302 INFO:tasks.workunit.client.0.smithi129.stderr:58362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.19 last_scrub_stamp 2024-03-25T23:11:25.302 INFO:tasks.workunit.client.0.smithi129.stderr:58362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.19 2024-03-25T23:11:25.302 INFO:tasks.workunit.client.0.smithi129.stderr:58362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:25.302 INFO:tasks.workunit.client.0.smithi129.stderr:58362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:25.302 INFO:tasks.workunit.client.0.smithi129.stderr:58362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2024-03-25T23:11:25.302 INFO:tasks.workunit.client.0.smithi129.stderr:58362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-25T23:09:44.508877+0000 '>' '' 2024-03-25T23:11:25.302 INFO:tasks.workunit.client.0.smithi129.stderr:58362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-25T23:11:25.302 INFO:tasks.workunit.client.0.smithi129.stderr:58362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-25T23:11:25.302 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:11:25.338 INFO:tasks.workunit.client.0.smithi129.stderr:58316: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.13 2024-03-25T23:11:25.338 INFO:tasks.workunit.client.0.smithi129.stderr:58316: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.13 2024-03-25T23:11:25.339 INFO:tasks.workunit.client.0.smithi129.stderr:58316: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.13 2024-03-25T23:11:25.339 INFO:tasks.workunit.client.0.smithi129.stderr:58316: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:25.339 INFO:tasks.workunit.client.0.smithi129.stderr:58316: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:25.339 INFO:tasks.workunit.client.0.smithi129.stderr:58316: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2024-03-25T23:11:25.339 INFO:tasks.workunit.client.0.smithi129.stderr:58316: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:25.339 INFO:tasks.workunit.client.0.smithi129.stderr:58316: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-25T23:11:25.339 INFO:tasks.workunit.client.0.smithi129.stderr:58316: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.13 query 2024-03-25T23:11:25.339 INFO:tasks.workunit.client.0.smithi129.stderr:58316: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-25T23:11:25.339 INFO:tasks.workunit.client.0.smithi129.stderr:58316: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-25T23:11:25.339 INFO:tasks.workunit.client.0.smithi129.stderr:58316: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-25T23:11:25.339 INFO:tasks.workunit.client.0.smithi129.stderr:58316: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.13 2024-03-25T23:11:25.339 INFO:tasks.workunit.client.0.smithi129.stderr:58316: instructing pg 1.13 on osd.4 to scrub 2024-03-25T23:11:25.339 INFO:tasks.workunit.client.0.smithi129.stderr:58316: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-25T23:11:25.339 INFO:tasks.workunit.client.0.smithi129.stderr:58316: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-25T23:11:25.340 INFO:tasks.workunit.client.0.smithi129.stderr:58316: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.13") | .state' 2024-03-25T23:11:25.340 INFO:tasks.workunit.client.0.smithi129.stderr:58316: dumped pgs 2024-03-25T23:11:25.340 INFO:tasks.workunit.client.0.smithi129.stderr:58316: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.13 2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:25.340 INFO:tasks.workunit.client.0.smithi129.stderr:58316: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.13 2024-03-25T23:11:25.340 INFO:tasks.workunit.client.0.smithi129.stderr:58316: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-25T23:09:44.508877+0000 2024-03-25T23:11:25.340 INFO:tasks.workunit.client.0.smithi129.stderr:58316: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-25T23:11:25.340 INFO:tasks.workunit.client.0.smithi129.stderr:58316: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-25T23:11:25.340 INFO:tasks.workunit.client.0.smithi129.stderr:58316: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-25T23:11:25.340 INFO:tasks.workunit.client.0.smithi129.stderr:58316: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-25T23:11:25.340 INFO:tasks.workunit.client.0.smithi129.stderr:58316: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-25T23:11:25.340 INFO:tasks.workunit.client.0.smithi129.stderr:58316: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.13 last_scrub_stamp 2024-03-25T23:11:25.340 INFO:tasks.workunit.client.0.smithi129.stderr:58316: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.13 2024-03-25T23:11:25.340 INFO:tasks.workunit.client.0.smithi129.stderr:58316: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:11:25.340 INFO:tasks.workunit.client.0.smithi129.stderr:58316: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:11:25.341 INFO:tasks.workunit.client.0.smithi129.stderr:58316: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2024-03-25T23:11:25.341 INFO:tasks.workunit.client.0.smithi129.stderr:58316: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-25T23:09:44.508877+0000 '>' '' 2024-03-25T23:11:25.341 INFO:tasks.workunit.client.0.smithi129.stderr:58316: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-25T23:11:25.341 INFO:tasks.workunit.client.0.smithi129.stderr:58316: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-25T23:11:25.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:11:25.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-25T23:11:25.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-25T23:11:25.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-25T23:11:25.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-25T23:11:25.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58313 2024-03-25T23:11:25.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-25T23:11:25.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-25T23:11:25.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-25T23:11:25.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-25T23:11:25.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58319 2024-03-25T23:11:25.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-25T23:11:25.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-25T23:11:25.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-25T23:11:25.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-25T23:11:25.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58337 2024-03-25T23:11:25.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-25T23:11:25.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-25T23:11:25.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-25T23:11:25.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-25T23:11:25.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58341 2024-03-25T23:11:25.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-25T23:11:25.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-25T23:11:25.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-25T23:11:25.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-25T23:11:25.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58348 2024-03-25T23:11:25.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-25T23:11:25.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-25T23:11:25.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-25T23:11:25.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-25T23:11:25.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58357 2024-03-25T23:11:25.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-25T23:11:25.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-25T23:11:25.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-25T23:11:25.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-25T23:11:25.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58369 2024-03-25T23:11:25.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-25T23:11:25.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-25T23:11:25.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-25T23:11:25.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-25T23:11:25.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58380 2024-03-25T23:11:25.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-25T23:11:25.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-25T23:11:25.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-25T23:11:25.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-25T23:11:25.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58393 2024-03-25T23:11:25.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-25T23:11:25.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-25T23:11:25.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-25T23:11:25.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-25T23:11:25.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58412 2024-03-25T23:11:25.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-25T23:11:25.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-25T23:11:25.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-25T23:11:25.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-25T23:11:25.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58417 2024-03-25T23:11:25.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=2 2024-03-25T23:11:25.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 2 -eq 2 2024-03-25T23:11:25.345 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: expr 1 + 1 2024-03-25T23:11:25.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: recov_scrub_count=2 2024-03-25T23:11:25.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-25T23:11:25.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58432 2024-03-25T23:11:25.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-25T23:11:25.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-25T23:11:25.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-25T23:11:25.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:227: wait_background_check: eval 'pids='\'''\''' 2024-03-25T23:11:25.346 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:227: wait_background_check: pids= 2024-03-25T23:11:25.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:229: wait_background_check: return 0 2024-03-25T23:11:25.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:307: TEST_recovery_scrub_2: return_code=0 2024-03-25T23:11:25.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:308: TEST_recovery_scrub_2: '[' 0 -ne 0 ']' 2024-03-25T23:11:25.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:310: TEST_recovery_scrub_2: ERRORS=0 2024-03-25T23:11:25.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:311: TEST_recovery_scrub_2: test 2 -eq 0 2024-03-25T23:11:25.347 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: find td/osd-recovery-scrub 2024-03-25T23:11:25.347 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: grep '[^/]*\.pid' 2024-03-25T23:11:25.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: pidfile='td/osd-recovery-scrub/osd.5.pid 2024-03-25T23:11:25.347 INFO:tasks.workunit.client.0.smithi129.stderr:td/osd-recovery-scrub/osd.2.pid 2024-03-25T23:11:25.347 INFO:tasks.workunit.client.0.smithi129.stderr:td/osd-recovery-scrub/osd.3.pid 2024-03-25T23:11:25.347 INFO:tasks.workunit.client.0.smithi129.stderr:td/osd-recovery-scrub/mgr.x.pid 2024-03-25T23:11:25.347 INFO:tasks.workunit.client.0.smithi129.stderr:td/osd-recovery-scrub/osd.7.pid 2024-03-25T23:11:25.347 INFO:tasks.workunit.client.0.smithi129.stderr:td/osd-recovery-scrub/osd.4.pid 2024-03-25T23:11:25.347 INFO:tasks.workunit.client.0.smithi129.stderr:td/osd-recovery-scrub/mon.a.pid 2024-03-25T23:11:25.347 INFO:tasks.workunit.client.0.smithi129.stderr:td/osd-recovery-scrub/osd.1.pid 2024-03-25T23:11:25.347 INFO:tasks.workunit.client.0.smithi129.stderr:td/osd-recovery-scrub/osd.0.pid 2024-03-25T23:11:25.347 INFO:tasks.workunit.client.0.smithi129.stderr:td/osd-recovery-scrub/osd.6.pid' 2024-03-25T23:11:25.347 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:318: TEST_recovery_scrub_2: cat td/osd-recovery-scrub/osd.5.pid td/osd-recovery-scrub/osd.2.pid td/osd-recovery-scrub/osd.3.pid td/osd-recovery-scrub/mgr.x.pid td/osd-recovery-scrub/osd.7.pid td/osd-recovery-scrub/osd.4.pid td/osd-recovery-scrub/mon.a.pid td/osd-recovery-scrub/osd.1.pid td/osd-recovery-scrub/osd.0.pid td/osd-recovery-scrub/osd.6.pid 2024-03-25T23:11:25.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:318: TEST_recovery_scrub_2: pid='53183 2024-03-25T23:11:25.348 INFO:tasks.workunit.client.0.smithi129.stderr:49053 2024-03-25T23:11:25.348 INFO:tasks.workunit.client.0.smithi129.stderr:50431 2024-03-25T23:11:25.348 INFO:tasks.workunit.client.0.smithi129.stderr:45358 2024-03-25T23:11:25.348 INFO:tasks.workunit.client.0.smithi129.stderr:55935 2024-03-25T23:11:25.348 INFO:tasks.workunit.client.0.smithi129.stderr:51807 2024-03-25T23:11:25.348 INFO:tasks.workunit.client.0.smithi129.stderr:45274 2024-03-25T23:11:25.348 INFO:tasks.workunit.client.0.smithi129.stderr:47677 2024-03-25T23:11:25.348 INFO:tasks.workunit.client.0.smithi129.stderr:46301 2024-03-25T23:11:25.348 INFO:tasks.workunit.client.0.smithi129.stderr:54559' 2024-03-25T23:11:25.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:319: TEST_recovery_scrub_2: kill -0 53183 49053 50431 45358 55935 51807 45274 47677 46301 54559 2024-03-25T23:11:25.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:327: TEST_recovery_scrub_2: kill_daemons td/osd-recovery-scrub 2024-03-25T23:11:25.348 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:11:25.348 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:11:25.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:11:25.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:11:25.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:11:26.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:11:26.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:329: TEST_recovery_scrub_2: declare -a err_strings 2024-03-25T23:11:26.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:330: TEST_recovery_scrub_2: err_strings[0]='not scheduling scrubs due to active recovery' 2024-03-25T23:11:26.651 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: expr 8 - 1 2024-03-25T23:11:26.652 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: seq 0 7 2024-03-25T23:11:26.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-25T23:11:26.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.0.log 2024-03-25T23:11:26.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-25T23:11:26.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.1.log 2024-03-25T23:11:26.666 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-25T23:11:26.666 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.2.log 2024-03-25T23:11:26.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-25T23:11:26.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.3.log 2024-03-25T23:11:26.675 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-25T23:11:26.675 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.4.log 2024-03-25T23:11:26.680 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-25T23:11:26.680 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.5.log 2024-03-25T23:11:26.685 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-25T23:11:26.685 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.6.log 2024-03-25T23:11:26.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-25T23:11:26.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.7.log 2024-03-25T23:11:26.693 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:336: TEST_recovery_scrub_2: for err_string in "${err_strings[@]}" 2024-03-25T23:11:26.693 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:338: TEST_recovery_scrub_2: found=false 2024-03-25T23:11:26.693 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: expr 8 - 1 2024-03-25T23:11:26.694 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: seq 0 7 2024-03-25T23:11:26.695 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-25T23:11:26.695 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.0.log 2024-03-25T23:11:26.700 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-25T23:11:26.700 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.1.log 2024-03-25T23:11:26.705 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-25T23:11:26.705 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.2.log 2024-03-25T23:11:26.708 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-25T23:11:26.708 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.3.log 2024-03-25T23:11:26.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-25T23:11:26.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.4.log 2024-03-25T23:11:26.718 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-25T23:11:26.718 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.5.log 2024-03-25T23:11:26.721 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-25T23:11:26.721 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.6.log 2024-03-25T23:11:26.725 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-25T23:11:26.725 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.7.log 2024-03-25T23:11:26.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:346: TEST_recovery_scrub_2: '[' false = true ']' 2024-03-25T23:11:26.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:352: TEST_recovery_scrub_2: teardown td/osd-recovery-scrub 2024-03-25T23:11:26.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-03-25T23:11:26.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T23:11:26.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-03-25T23:11:26.729 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:11:26.730 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:11:26.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:11:26.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:11:26.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:11:26.739 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:11:26.740 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T23:11:26.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T23:11:26.741 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T23:11:26.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T23:11:26.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T23:11:26.742 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T23:11:26.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:11:26.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T23:11:26.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T23:11:26.743 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:11:26.744 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T23:11:26.745 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T23:11:26.745 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-03-25T23:11:26.771 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T23:11:26.771 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:11:26.772 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36561 2024-03-25T23:11:26.772 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36561 2024-03-25T23:11:26.772 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T23:11:26.773 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T23:11:26.773 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:354: TEST_recovery_scrub_2: '[' 0 '!=' 0 ']' 2024-03-25T23:11:26.773 INFO:tasks.workunit.client.0.smithi129.stdout:TEST PASSED 2024-03-25T23:11:26.775 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:360: TEST_recovery_scrub_2: echo 'TEST PASSED' 2024-03-25T23:11:26.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:361: TEST_recovery_scrub_2: return 0 2024-03-25T23:11:26.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-03-25T23:11:26.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-recovery-scrub 0 2024-03-25T23:11:26.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-03-25T23:11:26.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-03-25T23:11:26.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-03-25T23:11:26.776 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:11:26.776 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:11:26.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:11:26.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:11:26.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:11:26.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:11:26.777 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T23:11:26.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T23:11:26.778 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T23:11:26.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T23:11:26.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T23:11:26.779 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T23:11:26.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:11:26.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T23:11:26.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T23:11:26.779 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:11:26.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T23:11:26.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-03-25T23:11:26.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-03-25T23:11:26.781 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T23:11:26.782 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:11:26.782 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36561 2024-03-25T23:11:26.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36561 2024-03-25T23:11:26.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T23:11:26.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T23:11:26.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-03-25T23:11:26.784 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-25T23:11:26.784 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-25T23:11:26.833 INFO:tasks.workunit:Running workunit scrub/osd-scrub-dump.sh... 2024-03-25T23:11:26.834 DEBUG:teuthology.orchestra.run.smithi129:workunit test scrub/osd-scrub-dump.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4354a20109fd25703f6611a1a8f196608a256149 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh 2024-03-25T23:11:26.887 INFO:tasks.workunit.client.0.smithi129.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-03-25T23:11:26.892 INFO:tasks.workunit.client.0.smithi129.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-03-25T23:11:26.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-03-25T23:11:26.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-03-25T23:11:26.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-03-25T23:11:26.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-03-25T23:11:26.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-03-25T23:11:26.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-03-25T23:11:26.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-03-25T23:11:26.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-03-25T23:11:26.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-scrub-dump 2024-03-25T23:11:26.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:30: run: echo 'This test is disabled' 2024-03-25T23:11:26.893 INFO:tasks.workunit.client.0.smithi129.stdout:This test is disabled 2024-03-25T23:11:26.893 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:31: run: return 0 2024-03-25T23:11:26.893 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-03-25T23:11:26.893 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-scrub-dump 0 2024-03-25T23:11:26.893 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-dump 2024-03-25T23:11:26.894 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-03-25T23:11:26.894 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-dump KILL 2024-03-25T23:11:26.894 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:11:26.894 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:11:26.894 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:11:26.894 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:11:26.894 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:11:26.895 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:11:26.895 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T23:11:26.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T23:11:26.896 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T23:11:26.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T23:11:26.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T23:11:26.897 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T23:11:26.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:11:26.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T23:11:26.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T23:11:26.899 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:11:26.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T23:11:26.900 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-03-25T23:11:26.900 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-dump 2024-03-25T23:11:26.901 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T23:11:26.901 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:11:26.901 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61612 2024-03-25T23:11:26.902 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61612 2024-03-25T23:11:26.903 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T23:11:26.903 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T23:11:26.903 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-03-25T23:11:26.903 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-25T23:11:26.903 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-25T23:11:26.958 INFO:tasks.workunit:Running workunit scrub/osd-scrub-repair.sh... 2024-03-25T23:11:26.958 DEBUG:teuthology.orchestra.run.smithi129:workunit test scrub/osd-scrub-repair.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4354a20109fd25703f6611a1a8f196608a256149 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh 2024-03-25T23:11:27.011 INFO:tasks.workunit.client.0.smithi129.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh 2024-03-25T23:11:27.011 INFO:tasks.workunit.client.0.smithi129.stderr:++ TIMEOUT=300 2024-03-25T23:11:27.011 INFO:tasks.workunit.client.0.smithi129.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2024-03-25T23:11:27.011 INFO:tasks.workunit.client.0.smithi129.stderr:++ MAX_TIMEOUT=15 2024-03-25T23:11:27.011 INFO:tasks.workunit.client.0.smithi129.stderr:++ PG_NUM=4 2024-03-25T23:11:27.011 INFO:tasks.workunit.client.0.smithi129.stderr:++ TMPDIR=/tmp 2024-03-25T23:11:27.011 INFO:tasks.workunit.client.0.smithi129.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2024-03-25T23:11:27.011 INFO:tasks.workunit.client.0.smithi129.stderr:++ TESTDIR=/home/ubuntu/cephtest 2024-03-25T23:11:27.012 INFO:tasks.workunit.client.0.smithi129.stderr:++ type xmlstarlet 2024-03-25T23:11:27.012 INFO:tasks.workunit.client.0.smithi129.stderr:++ XMLSTARLET=xmlstarlet 2024-03-25T23:11:27.012 INFO:tasks.workunit.client.0.smithi129.stderr:+++ uname 2024-03-25T23:11:27.012 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' Linux = FreeBSD ']' 2024-03-25T23:11:27.012 INFO:tasks.workunit.client.0.smithi129.stderr:++ SED=sed 2024-03-25T23:11:27.012 INFO:tasks.workunit.client.0.smithi129.stderr:++ AWK=awk 2024-03-25T23:11:27.013 INFO:tasks.workunit.client.0.smithi129.stderr:+++ stty -a 2024-03-25T23:11:27.013 INFO:tasks.workunit.client.0.smithi129.stderr:+++ head -1 2024-03-25T23:11:27.013 INFO:tasks.workunit.client.0.smithi129.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2024-03-25T23:11:27.013 INFO:tasks.workunit.client.0.smithi129.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-03-25T23:11:27.014 INFO:tasks.workunit.client.0.smithi129.stderr:++ termwidth= 2024-03-25T23:11:27.014 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2024-03-25T23:11:27.014 INFO:tasks.workunit.client.0.smithi129.stderr:++ DIFFCOLOPTS='-y ' 2024-03-25T23:11:27.014 INFO:tasks.workunit.client.0.smithi129.stderr:++ KERNCORE=kernel.core_pattern 2024-03-25T23:11:27.014 INFO:tasks.workunit.client.0.smithi129.stderr:++ EXTRA_OPTS= 2024-03-25T23:11:27.017 INFO:tasks.workunit.client.0.smithi129.stderr:++ test '' = TESTS 2024-03-25T23:11:27.017 INFO:tasks.workunit.client.0.smithi129.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh 2024-03-25T23:11:27.018 INFO:tasks.workunit.client.0.smithi129.stderr:++ uname 2024-03-25T23:11:27.018 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' Linux = FreeBSD ']' 2024-03-25T23:11:27.019 INFO:tasks.workunit.client.0.smithi129.stderr:+ use_ec_overwrite=true 2024-03-25T23:11:27.019 INFO:tasks.workunit.client.0.smithi129.stderr:+ getjson=no 2024-03-25T23:11:27.019 INFO:tasks.workunit.client.0.smithi129.stderr:+ jqfilter='def walk(f): 2024-03-25T23:11:27.019 INFO:tasks.workunit.client.0.smithi129.stderr: . as $in 2024-03-25T23:11:27.019 INFO:tasks.workunit.client.0.smithi129.stderr: | if type == "object" then 2024-03-25T23:11:27.019 INFO:tasks.workunit.client.0.smithi129.stderr: reduce keys[] as $key 2024-03-25T23:11:27.019 INFO:tasks.workunit.client.0.smithi129.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-25T23:11:27.019 INFO:tasks.workunit.client.0.smithi129.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-25T23:11:27.019 INFO:tasks.workunit.client.0.smithi129.stderr: else f 2024-03-25T23:11:27.020 INFO:tasks.workunit.client.0.smithi129.stderr: end; 2024-03-25T23:11:27.020 INFO:tasks.workunit.client.0.smithi129.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-25T23:11:27.020 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-25T23:11:27.020 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-25T23:11:27.020 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-25T23:11:27.020 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-03-25T23:11:27.020 INFO:tasks.workunit.client.0.smithi129.stderr:+ sortkeys='import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-25T23:11:27.024 INFO:tasks.workunit.client.0.smithi129.stderr:+ main osd-scrub-repair 2024-03-25T23:11:27.025 INFO:tasks.workunit.client.0.smithi129.stderr:+ local dir=td/osd-scrub-repair 2024-03-25T23:11:27.025 INFO:tasks.workunit.client.0.smithi129.stderr:+ shift 2024-03-25T23:11:27.025 INFO:tasks.workunit.client.0.smithi129.stderr:+ shopt -s -o xtrace 2024-03-25T23:11:27.025 INFO:tasks.workunit.client.0.smithi129.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-03-25T23:11:27.025 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-03-25T23:11:27.025 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-03-25T23:11:27.025 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-03-25T23:11:27.026 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-03-25T23:11:27.026 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-03-25T23:11:27.026 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-03-25T23:11:27.026 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-03-25T23:11:27.026 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-03-25T23:11:27.026 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-scrub-repair 2024-03-25T23:11:27.026 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:52: run: local dir=td/osd-scrub-repair 2024-03-25T23:11:27.026 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:53: run: shift 2024-03-25T23:11:27.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:55: run: export CEPH_MON=127.0.0.1:7107 2024-03-25T23:11:27.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:55: run: CEPH_MON=127.0.0.1:7107 2024-03-25T23:11:27.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:56: run: export CEPH_ARGS 2024-03-25T23:11:27.027 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:57: run: uuidgen 2024-03-25T23:11:27.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:57: run: CEPH_ARGS+='--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none ' 2024-03-25T23:11:27.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:58: run: CEPH_ARGS+='--mon-host=127.0.0.1:7107 ' 2024-03-25T23:11:27.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:59: run: CEPH_ARGS+='--osd-skip-data-digest=false ' 2024-03-25T23:11:27.028 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:61: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-03-25T23:11:27.028 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: set 2024-03-25T23:11:27.028 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-03-25T23:11:27.030 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: local 'funcs=TEST_allow_repair_during_recovery 2024-03-25T23:11:27.030 INFO:tasks.workunit.client.0.smithi129.stderr:TEST_auto_repair_bluestore_basic 2024-03-25T23:11:27.030 INFO:tasks.workunit.client.0.smithi129.stderr:TEST_auto_repair_bluestore_failed 2024-03-25T23:11:27.030 INFO:tasks.workunit.client.0.smithi129.stderr:TEST_auto_repair_bluestore_failed_norecov 2024-03-25T23:11:27.030 INFO:tasks.workunit.client.0.smithi129.stderr:TEST_auto_repair_bluestore_scrub 2024-03-25T23:11:27.030 INFO:tasks.workunit.client.0.smithi129.stderr:TEST_auto_repair_bluestore_tag 2024-03-25T23:11:27.031 INFO:tasks.workunit.client.0.smithi129.stderr:TEST_auto_repair_erasure_coded_appends 2024-03-25T23:11:27.031 INFO:tasks.workunit.client.0.smithi129.stderr:TEST_auto_repair_erasure_coded_overwrites 2024-03-25T23:11:27.031 INFO:tasks.workunit.client.0.smithi129.stderr:TEST_corrupt_and_repair_jerasure_appends 2024-03-25T23:11:27.031 INFO:tasks.workunit.client.0.smithi129.stderr:TEST_corrupt_and_repair_jerasure_overwrites 2024-03-25T23:11:27.031 INFO:tasks.workunit.client.0.smithi129.stderr:TEST_corrupt_and_repair_lrc_appends 2024-03-25T23:11:27.031 INFO:tasks.workunit.client.0.smithi129.stderr:TEST_corrupt_and_repair_lrc_overwrites 2024-03-25T23:11:27.031 INFO:tasks.workunit.client.0.smithi129.stderr:TEST_corrupt_and_repair_replicated 2024-03-25T23:11:27.031 INFO:tasks.workunit.client.0.smithi129.stderr:TEST_corrupt_scrub_erasure_appends 2024-03-25T23:11:27.031 INFO:tasks.workunit.client.0.smithi129.stderr:TEST_corrupt_scrub_erasure_overwrites 2024-03-25T23:11:27.031 INFO:tasks.workunit.client.0.smithi129.stderr:TEST_corrupt_scrub_replicated 2024-03-25T23:11:27.031 INFO:tasks.workunit.client.0.smithi129.stderr:TEST_corrupt_snapset_scrub_rep 2024-03-25T23:11:27.031 INFO:tasks.workunit.client.0.smithi129.stderr:TEST_list_missing_erasure_coded_appends 2024-03-25T23:11:27.031 INFO:tasks.workunit.client.0.smithi129.stderr:TEST_list_missing_erasure_coded_overwrites 2024-03-25T23:11:27.031 INFO:tasks.workunit.client.0.smithi129.stderr:TEST_periodic_scrub_replicated 2024-03-25T23:11:27.031 INFO:tasks.workunit.client.0.smithi129.stderr:TEST_repair_stats 2024-03-25T23:11:27.032 INFO:tasks.workunit.client.0.smithi129.stderr:TEST_repair_stats_ec 2024-03-25T23:11:27.032 INFO:tasks.workunit.client.0.smithi129.stderr:TEST_request_scrub_priority 2024-03-25T23:11:27.032 INFO:tasks.workunit.client.0.smithi129.stderr:TEST_scrub_warning 2024-03-25T23:11:27.032 INFO:tasks.workunit.client.0.smithi129.stderr:TEST_skip_non_repair_during_recovery 2024-03-25T23:11:27.032 INFO:tasks.workunit.client.0.smithi129.stderr:TEST_unfound_erasure_coded_appends 2024-03-25T23:11:27.032 INFO:tasks.workunit.client.0.smithi129.stderr:TEST_unfound_erasure_coded_overwrites' 2024-03-25T23:11:27.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-25T23:11:27.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-25T23:11:27.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-25T23:11:27.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-25T23:11:27.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-25T23:11:27.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T23:11:27.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-25T23:11:27.032 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:11:27.032 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:11:27.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:11:27.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:11:27.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:11:27.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:11:27.033 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T23:11:27.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T23:11:27.034 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T23:11:27.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T23:11:27.035 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T23:11:27.035 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T23:11:27.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:11:27.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T23:11:27.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T23:11:27.037 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:11:27.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T23:11:27.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T23:11:27.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-25T23:11:27.040 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T23:11:27.040 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:11:27.040 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:11:27.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-25T23:11:27.041 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T23:11:27.041 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T23:11:27.041 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-25T23:11:27.043 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-25T23:11:27.043 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:11:27.043 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:11:27.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61636 2024-03-25T23:11:27.047 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-25T23:11:27.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-03-25T23:11:27.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-03-25T23:11:27.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-25T23:11:27.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-25T23:11:27.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_allow_repair_during_recovery td/osd-scrub-repair 2024-03-25T23:11:27.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:114: TEST_allow_repair_during_recovery: local dir=td/osd-scrub-repair 2024-03-25T23:11:27.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:115: TEST_allow_repair_during_recovery: local poolname=rbd 2024-03-25T23:11:27.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:117: TEST_allow_repair_during_recovery: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-03-25T23:11:27.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-25T23:11:27.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-25T23:11:27.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-25T23:11:27.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-25T23:11:27.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-25T23:11:27.049 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-03-25T23:11:27.544 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-25T23:11:27.545 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:11:27.545 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:11:27.545 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:11:27.545 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:11:27.545 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:11:27.546 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:11:27.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-03-25T23:11:27.765 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-25T23:11:27.766 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-25T23:11:27.766 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T23:11:27.766 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T23:11:27.766 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-25T23:11:27.767 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-25T23:11:27.767 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T23:11:27.767 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T23:11:27.767 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T23:11:27.768 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T23:11:27.768 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:11:27.768 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:11:27.768 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-25T23:11:27.768 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T23:11:27.768 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get fsid 2024-03-25T23:11:27.878 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-25T23:11:27.878 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T23:11:27.878 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T23:11:27.878 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-25T23:11:27.880 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-25T23:11:27.880 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T23:11:27.880 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T23:11:27.880 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T23:11:27.880 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T23:11:27.880 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:11:27.880 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:11:27.880 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-25T23:11:27.880 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T23:11:27.880 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get mon_host 2024-03-25T23:11:27.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:118: TEST_allow_repair_during_recovery: run_mgr td/osd-scrub-repair x 2024-03-25T23:11:27.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-25T23:11:27.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-25T23:11:27.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-25T23:11:27.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-25T23:11:27.980 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-25T23:11:27.980 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-25T23:11:28.847 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-25T23:11:28.847 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:11:28.847 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:11:28.847 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:11:28.847 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:11:28.847 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:11:28.848 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:11:28.848 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T23:11:28.849 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T23:11:28.886 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:119: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 0 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-03-25T23:11:28.886 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:11:28.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:11:28.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-25T23:11:28.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:11:28.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-25T23:11:28.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:11:28.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:11:28.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:11:28.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:11:28.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-25T23:11:28.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-25T23:11:28.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:11:28.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:11:28.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:11:28.888 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:11:28.888 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:11:28.888 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:11:28.888 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:11:28.888 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:11:28.888 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:11:28.888 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:11:28.888 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:11:28.888 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:11:28.888 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:11:28.888 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:11:28.888 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:11:28.888 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:11:28.888 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:11:28.889 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:11:28.889 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:11:28.889 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:11:28.889 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:11:28.889 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-03-25T23:11:28.889 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-25T23:11:28.889 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:11:28.889 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=104d42e5-c42e-4474-bab0-1c1b6a377418 2024-03-25T23:11:28.890 INFO:tasks.workunit.client.0.smithi129.stdout:add osd0 104d42e5-c42e-4474-bab0-1c1b6a377418 2024-03-25T23:11:28.890 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 104d42e5-c42e-4474-bab0-1c1b6a377418' 2024-03-25T23:11:28.890 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:11:28.906 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCgBAJmprVINhAACbFdcaiSs/jj3ZLO0O74nQ== 2024-03-25T23:11:28.906 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCgBAJmprVINhAACbFdcaiSs/jj3ZLO0O74nQ=="}' 2024-03-25T23:11:28.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 104d42e5-c42e-4474-bab0-1c1b6a377418 -i td/osd-scrub-repair/0/new.json 2024-03-25T23:11:29.175 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:11:29.193 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-25T23:11:29.195 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQCgBAJmprVINhAACbFdcaiSs/jj3ZLO0O74nQ== --osd-uuid 104d42e5-c42e-4474-bab0-1c1b6a377418 2024-03-25T23:11:29.232 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:11:29.234+0000 7f66109bb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:11:29.242 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:11:29.242+0000 7f66109bb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:11:29.244 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:11:29.242+0000 7f66109bb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:11:29.244 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:11:29.246+0000 7f66109bb7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-25T23:11:29.244 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:11:29.246+0000 7f66109bb7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-25T23:11:29.245 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:11:29.246+0000 7f66109bb7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-25T23:11:29.343 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:11:29.342+0000 7f66109bb7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-25T23:11:35.875 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-25T23:11:35.875 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:11:35.877 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd0 key to auth repository 2024-03-25T23:11:35.877 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-25T23:11:35.877 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:11:36.255 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.0 2024-03-25T23:11:36.255 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-25T23:11:36.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-03-25T23:11:36.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:11:36.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:11:36.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:11:36.288 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:11:36.286+0000 7f39a2d517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:11:36.317 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:11:36.318+0000 7f39a2d517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:11:36.319 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:11:36.318+0000 7f39a2d517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:11:36.584 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-25T23:11:36.584 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:11:36.584 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T23:11:36.584 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:11:36.584 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:11:36.584 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:11:36.584 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:11:36.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:11:36.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:11:36.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:11:36.876 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:11:36.874+0000 7f39a2d517c0 -1 Falling back to public interface 2024-03-25T23:11:36.924 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:11:37.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:11:37.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:11:37.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:11:37.926 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:11:37.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:11:37.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:11:38.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:11:39.241 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:11:39.241 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:11:39.242 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:11:39.242 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:11:39.242 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:11:39.242 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:11:39.479 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:11:39.478+0000 7f39a2d517c0 -1 osd.0 0 log_to_monitors true 2024-03-25T23:11:39.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:11:40.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:11:40.589 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:11:40.589 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:11:40.589 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:11:40.589 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:11:40.590 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:11:40.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:11:41.909 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:11:41.909 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:11:41.909 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:11:41.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:11:41.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:11:41.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:11:42.275 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:11:43.277 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:11:43.277 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:11:43.277 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:11:43.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:11:43.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:11:43.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:11:43.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:11:44.597 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:11:44.597 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:11:44.598 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:11:44.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:11:44.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:11:44.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:11:44.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:11:45.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:11:45.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:11:45.919 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:11:45.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:11:45.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:11:45.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:11:46.315 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:11:47.317 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:11:47.317 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:11:47.317 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:11:47.318 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:11:47.318 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:11:47.318 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:11:47.633 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:11:48.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:11:48.635 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:11:48.635 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:11:48.635 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:11:48.635 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:11:48.636 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:11:48.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:11:49.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:11:49.950 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:11:49.950 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:11:49.950 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:11:49.951 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:11:49.951 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:11:50.356 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:11:51.357 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:11:51.358 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:11:51.358 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:11:51.358 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:11:51.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:11:51.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:11:51.680 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:11:52.682 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:11:52.682 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:11:52.682 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:11:52.683 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:11:52.683 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:11:52.683 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:11:52.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:11:53.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:11:53.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:11:53.988 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:11:53.989 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:11:53.989 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:11:53.989 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:11:54.320 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:11:55.322 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:11:55.322 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:11:55.322 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:11:55.323 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:11:55.323 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:11:55.323 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:11:55.795 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:11:56.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:11:56.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:11:56.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:11:56.797 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:11:56.798 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:11:56.798 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:11:57.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:11:58.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:11:58.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:11:58.349 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:11:58.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:11:58.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:11:58.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:11:58.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:11:59.875 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:11:59.875 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:11:59.875 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:11:59.876 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:11:59.876 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:11:59.876 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:12:00.718 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:12:01.720 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:12:01.720 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:12:01.721 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:12:01.721 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:12:01.721 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:12:01.722 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:12:02.476 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:12:03.477 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:12:03.478 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:12:03.478 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:12:03.482 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:12:03.482 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:12:03.483 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:12:04.103 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:12:05.105 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:12:05.106 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:12:05.106 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-25T23:12:05.106 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-25T23:12:05.106 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:12:05.106 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:12:05.427 INFO:tasks.workunit.client.0.smithi129.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2075535642,v1:127.0.0.1:6803/2075535642] [v2:127.0.0.1:6804/2075535642,v1:127.0.0.1:6805/2075535642] exists,up 104d42e5-c42e-4474-bab0-1c1b6a377418 2024-03-25T23:12:05.429 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:12:05.430 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:12:05.430 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:12:05.430 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:122: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 1 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-03-25T23:12:05.430 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:12:05.430 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:12:05.430 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-25T23:12:05.430 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:12:05.430 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:12:05.430 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:12:05.430 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:12:05.430 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:12:05.430 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:12:05.430 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-25T23:12:05.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-25T23:12:05.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:12:05.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:12:05.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:12:05.431 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:12:05.431 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:12:05.431 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:12:05.433 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:12:05.433 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:12:05.433 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:12:05.433 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:12:05.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:12:05.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:12:05.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:12:05.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:12:05.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:12:05.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:12:05.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:12:05.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:12:05.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:12:05.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:12:05.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:12:05.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-03-25T23:12:05.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-25T23:12:05.434 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:12:05.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8976cf0a-9b81-4233-8ef6-5c9b582737e8 2024-03-25T23:12:05.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 8976cf0a-9b81-4233-8ef6-5c9b582737e8' 2024-03-25T23:12:05.435 INFO:tasks.workunit.client.0.smithi129.stdout:add osd1 8976cf0a-9b81-4233-8ef6-5c9b582737e8 2024-03-25T23:12:05.435 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:12:05.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDFBAJmsCU3GxAAT5WTuE+Fm9f90Ihb9zvr6g== 2024-03-25T23:12:05.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDFBAJmsCU3GxAAT5WTuE+Fm9f90Ihb9zvr6g=="}' 2024-03-25T23:12:05.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8976cf0a-9b81-4233-8ef6-5c9b582737e8 -i td/osd-scrub-repair/1/new.json 2024-03-25T23:12:05.911 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:12:05.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-25T23:12:05.932 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQDFBAJmsCU3GxAAT5WTuE+Fm9f90Ihb9zvr6g== --osd-uuid 8976cf0a-9b81-4233-8ef6-5c9b582737e8 2024-03-25T23:12:05.969 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:12:05.973+0000 7fb9d5ec37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:12:06.013 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:12:06.017+0000 7fb9d5ec37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:12:06.014 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:12:06.017+0000 7fb9d5ec37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:12:06.015 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:12:06.017+0000 7fb9d5ec37c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-25T23:12:06.015 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:12:06.017+0000 7fb9d5ec37c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-25T23:12:06.015 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:12:06.017+0000 7fb9d5ec37c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-25T23:12:06.114 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:12:06.117+0000 7fb9d5ec37c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-25T23:12:13.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-25T23:12:13.091 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:12:13.092 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd1 key to auth repository 2024-03-25T23:12:13.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-25T23:12:13.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:12:13.483 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.1 2024-03-25T23:12:13.483 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-25T23:12:13.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-03-25T23:12:13.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:12:13.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:12:13.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:12:13.518 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:12:13.521+0000 7fcbc13407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:12:13.521 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:12:13.525+0000 7fcbc13407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:12:13.522 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:12:13.525+0000 7fcbc13407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:12:13.793 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-25T23:12:13.793 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:12:13.793 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:12:13.794 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T23:12:13.794 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:12:13.794 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:12:13.794 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:12:13.794 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:12:13.794 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:12:13.794 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:12:14.059 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:12:14.061+0000 7fcbc13407c0 -1 Falling back to public interface 2024-03-25T23:12:14.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:12:15.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:12:15.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:12:15.114 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:12:15.114 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:12:15.114 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:12:15.114 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:12:15.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:12:16.421 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:12:16.421 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:12:16.421 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:12:16.421 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:12:16.422 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:12:16.422 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:12:16.683 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:12:16.685+0000 7fcbc13407c0 -1 osd.1 0 log_to_monitors true 2024-03-25T23:12:16.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:12:17.736 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:12:17.736 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:12:17.736 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:12:17.737 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:12:17.737 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:12:17.737 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:12:18.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:12:19.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:12:19.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:12:19.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:12:19.141 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:12:19.141 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:12:19.141 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:12:19.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:12:20.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:12:20.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:12:20.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:12:20.459 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:12:20.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:12:20.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:12:20.837 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:12:21.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:12:21.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:12:21.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:12:21.839 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:12:21.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:12:21.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:12:22.288 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:12:23.290 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:12:23.290 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:12:23.290 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:12:23.291 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:12:23.291 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:12:23.291 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:12:23.728 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:12:24.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:12:24.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:12:24.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:12:24.731 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:12:24.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:12:24.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:12:25.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:12:26.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:12:26.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:12:26.042 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:12:26.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:12:26.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:12:26.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:12:26.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:12:27.495 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:12:27.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:12:27.496 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:12:27.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:12:27.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:12:27.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:12:28.054 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:12:29.056 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:12:29.056 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:12:29.056 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:12:29.056 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:12:29.057 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:12:29.057 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:12:29.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:12:30.466 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:12:30.466 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:12:30.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:12:30.467 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:12:30.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:12:30.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:12:30.841 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:12:31.842 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:12:31.843 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:12:31.843 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:12:31.843 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:12:31.844 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:12:31.844 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:12:32.762 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:12:33.764 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:12:33.764 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:12:33.764 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:12:33.764 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:12:33.765 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:12:33.765 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:12:34.465 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:12:35.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:12:35.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:12:35.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:12:35.467 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:12:35.468 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:12:35.468 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:12:36.437 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:12:37.438 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:12:37.439 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:12:37.439 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:12:37.439 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:12:37.439 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:12:37.439 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:12:37.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:12:38.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:12:38.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:12:38.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:12:38.988 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:12:38.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:12:38.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:12:39.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:12:40.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:12:40.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:12:40.581 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:12:40.582 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:12:40.582 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:12:40.582 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:12:41.414 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:12:42.417 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:12:42.417 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:12:42.417 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:12:42.417 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:12:42.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:12:42.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:12:42.864 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:12:43.866 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:12:43.866 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:12:43.867 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-25T23:12:43.867 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-25T23:12:43.867 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:12:43.867 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:12:44.187 INFO:tasks.workunit.client.0.smithi129.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1525863271,v1:127.0.0.1:6811/1525863271] [v2:127.0.0.1:6812/1525863271,v1:127.0.0.1:6813/1525863271] exists,up 8976cf0a-9b81-4233-8ef6-5c9b582737e8 2024-03-25T23:12:44.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:12:44.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:12:44.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:12:44.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:125: TEST_allow_repair_during_recovery: create_rbd_pool 2024-03-25T23:12:44.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-25T23:12:44.512 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'rbd' does not exist 2024-03-25T23:12:44.529 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-25T23:12:44.529 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-25T23:12:44.979 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'rbd' created 2024-03-25T23:12:45.000 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T23:12:46.002 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-25T23:12:48.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:126: TEST_allow_repair_during_recovery: wait_for_clean 2024-03-25T23:12:48.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:12:48.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:12:48.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:12:48.458 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:12:48.459 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:12:48.459 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:12:48.459 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:12:48.459 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:12:48.459 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:12:48.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:12:48.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:12:48.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:12:48.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:12:48.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:12:48.581 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:12:48.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:12:48.900 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-25T23:12:48.900 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:12:48.900 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:12:48.900 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:12:49.075 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836491 2024-03-25T23:12:49.075 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836491 2024-03-25T23:12:49.075 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836491' 2024-03-25T23:12:49.075 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:12:49.076 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:12:49.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672966 2024-03-25T23:12:49.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672966 2024-03-25T23:12:49.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836491 1-42949672966' 2024-03-25T23:12:49.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:12:49.218 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836491 2024-03-25T23:12:49.218 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:12:49.218 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:12:49.219 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836491 2024-03-25T23:12:49.219 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:12:49.220 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836491 2024-03-25T23:12:49.220 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2024-03-25T23:12:49.221 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 21474836491 2024-03-25T23:12:49.221 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:12:49.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836491 -lt 21474836491 2024-03-25T23:12:49.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:12:49.515 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672966 2024-03-25T23:12:49.516 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:12:49.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:12:49.518 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672966 2024-03-25T23:12:49.518 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:12:49.519 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672966 2024-03-25T23:12:49.520 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 42949672966 2024-03-25T23:12:49.520 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-03-25T23:12:49.520 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:12:49.847 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672967 -lt 42949672966 2024-03-25T23:12:49.848 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:12:49.848 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:12:49.849 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:12:50.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-25T23:12:50.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:12:50.261 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:12:50.261 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:12:50.261 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:12:50.261 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:12:50.262 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:12:50.262 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:12:50.591 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T23:12:50.591 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:12:50.592 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:12:50.592 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:12:50.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-25T23:12:50.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:12:50.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:12:50.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:128: TEST_allow_repair_during_recovery: add_something td/osd-scrub-repair rbd 2024-03-25T23:12:50.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-25T23:12:50.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2024-03-25T23:12:50.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-03-25T23:12:50.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-25T23:12:50.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-25T23:12:50.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-25T23:12:51.366 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-25T23:12:51.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-25T23:12:51.944 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-25T23:12:51.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-25T23:12:51.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-25T23:12:51.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-25T23:12:52.293 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:129: TEST_allow_repair_during_recovery: get_not_primary rbd SOMETHING 2024-03-25T23:12:52.293 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=rbd 2024-03-25T23:12:52.294 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-25T23:12:52.294 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary rbd SOMETHING 2024-03-25T23:12:52.294 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-03-25T23:12:52.294 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-25T23:12:52.294 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-03-25T23:12:52.295 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-25T23:12:52.637 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-25T23:12:52.637 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map rbd SOMETHING 2024-03-25T23:12:52.637 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-25T23:12:52.983 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:129: TEST_allow_repair_during_recovery: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2024-03-25T23:12:52.983 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-25T23:12:52.983 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-03-25T23:12:52.983 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=0 2024-03-25T23:12:52.983 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-03-25T23:12:52.984 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:12:52.984 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:12:52.984 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-25T23:12:52.984 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:12:52.984 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-03-25T23:12:52.984 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:12:52.984 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:12:52.984 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-25T23:12:52.984 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:12:52.984 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-25T23:12:52.984 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:12:52.984 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:12:52.984 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:12:52.985 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:12:52.985 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:12:53.293 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:12:53.293 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-03-25T23:12:53.293 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:12:53.293 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:12:53.293 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-25T23:12:53.293 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:12:53.294 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-25T23:12:53.294 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-03-25T23:12:54.950 INFO:tasks.workunit.client.0.smithi129.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-03-25T23:12:55.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-25T23:12:55.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:12:55.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:12:55.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-25T23:12:55.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:12:55.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-25T23:12:55.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:12:55.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:12:55.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:12:55.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:12:55.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-25T23:12:55.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-25T23:12:55.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:12:55.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:12:55.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:12:55.485 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:12:55.485 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:12:55.485 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:12:55.487 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:12:55.487 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:12:55.487 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:12:55.487 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:12:55.487 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:12:55.487 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:12:55.487 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:12:55.487 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:12:55.487 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:12:55.487 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:12:55.487 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:12:55.487 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:12:55.487 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:12:55.488 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:12:55.488 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-25T23:12:55.488 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-25T23:12:55.488 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.0 2024-03-25T23:12:55.488 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:12:55.489 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-25T23:12:55.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-25T23:12:55.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:12:55.490 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:12:55.490 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:12:55.524 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:12:55.524+0000 7f4c64c687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:12:55.560 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:12:55.560+0000 7f4c64c687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:12:55.562 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:12:55.564+0000 7f4c64c687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:12:55.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-25T23:12:55.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:12:55.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T23:12:55.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:12:55.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:12:55.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:12:55.801 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:12:55.801 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-25T23:12:55.801 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:12:55.801 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:12:56.099 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:12:56.100+0000 7f4c64c687c0 -1 Falling back to public interface 2024-03-25T23:12:56.126 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:12:57.128 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:12:57.128 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:12:57.128 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:12:57.128 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:12:57.129 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:12:57.129 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:12:57.449 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:12:58.451 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:12:58.451 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:12:58.451 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:12:58.452 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:12:58.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:12:58.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:12:58.774 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:12:58.792 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:12:58.792+0000 7f4c64c687c0 -1 osd.0 21 log_to_monitors true 2024-03-25T23:12:59.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:12:59.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:12:59.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:12:59.776 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:12:59.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:12:59.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:13:00.178 INFO:tasks.workunit.client.0.smithi129.stderr:osd.0 up in weight 1 up_from 26 up_thru 26 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/3335014685,v1:127.0.0.1:6803/3335014685] [v2:127.0.0.1:6804/3335014685,v1:127.0.0.1:6805/3335014685] exists,up 104d42e5-c42e-4474-bab0-1c1b6a377418 2024-03-25T23:13:00.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:13:00.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:13:00.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:13:00.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:13:00.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:13:00.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:13:00.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:13:00.179 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:13:00.179 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:13:00.179 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:13:00.180 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:13:00.180 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:13:00.180 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:13:00.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:13:00.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:13:00.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:13:00.282 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:13:00.282 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:13:00.282 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:13:00.616 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:13:00.617 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-25T23:13:00.617 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:13:00.617 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:13:00.617 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:13:00.764 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149698 2024-03-25T23:13:00.764 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149698 2024-03-25T23:13:00.764 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149698' 2024-03-25T23:13:00.764 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:13:00.764 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:13:00.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672970 2024-03-25T23:13:00.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672970 2024-03-25T23:13:00.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149698 1-42949672970' 2024-03-25T23:13:00.980 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:13:00.980 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149698 2024-03-25T23:13:00.980 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:13:00.980 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:13:00.981 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149698 2024-03-25T23:13:00.981 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:13:00.982 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149698 2024-03-25T23:13:00.982 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149698' 2024-03-25T23:13:00.982 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 111669149698 2024-03-25T23:13:00.983 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:13:01.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 111669149698 2024-03-25T23:13:01.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:13:02.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:13:02.300 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:13:02.605 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149698 -lt 111669149698 2024-03-25T23:13:02.605 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:13:02.606 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672970 2024-03-25T23:13:02.606 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:13:02.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:13:02.608 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672970 2024-03-25T23:13:02.608 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:13:02.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672970 2024-03-25T23:13:02.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-03-25T23:13:02.609 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 42949672970 2024-03-25T23:13:02.610 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:13:02.947 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672970 -lt 42949672970 2024-03-25T23:13:02.948 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:13:02.948 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:13:02.949 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:13:03.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-25T23:13:03.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:13:03.344 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:13:03.344 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:13:03.344 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:13:03.344 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:13:03.344 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:13:03.344 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:13:03.714 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T23:13:03.714 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:13:03.715 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:13:03.715 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:13:04.082 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-25T23:13:04.082 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:13:04.082 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:13:04.082 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-03-25T23:13:04.082 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-03-25T23:13:04.082 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-25T23:13:04.082 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-03-25T23:13:04.083 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-25T23:13:04.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-03-25T23:13:04.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-03-25T23:13:04.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.3 2024-03-25T23:13:04.435 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.3 2024-03-25T23:13:04.435 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-25T23:13:04.435 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:13:04.436 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-25T23:13:04.436 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:13:04.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-25T23:12:44.920076+0000 2024-03-25T23:13:04.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.3 2024-03-25T23:13:05.060 INFO:tasks.workunit.client.0.smithi129.stderr:instructing pg 1.3 on osd.1 to repair 2024-03-25T23:13:05.075 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.3 2024-03-25T23:12:44.920076+0000 2024-03-25T23:13:05.076 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.3 2024-03-25T23:13:05.076 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-25T23:12:44.920076+0000 2024-03-25T23:13:05.076 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-25T23:13:05.076 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-25T23:13:05.076 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:13:05.077 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-25T23:13:05.077 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-25T23:13:05.077 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:13:05.077 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:13:05.077 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-25T23:13:05.378 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:12:44.920076+0000 '>' 2024-03-25T23:12:44.920076+0000 2024-03-25T23:13:05.378 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:13:06.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:13:06.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:13:06.380 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-25T23:13:06.380 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-25T23:13:06.380 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:13:06.380 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:13:06.381 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-25T23:13:06.690 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:12:44.920076+0000 '>' 2024-03-25T23:12:44.920076+0000 2024-03-25T23:13:06.690 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:13:07.691 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:13:07.691 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:13:07.692 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-25T23:13:07.692 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-25T23:13:07.692 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:13:07.693 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:13:07.693 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-25T23:13:08.003 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:12:44.920076+0000 '>' 2024-03-25T23:12:44.920076+0000 2024-03-25T23:13:08.003 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:13:09.005 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:13:09.005 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:13:09.005 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-25T23:13:09.006 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-25T23:13:09.006 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:13:09.006 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:13:09.006 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-25T23:13:09.306 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:12:44.920076+0000 '>' 2024-03-25T23:12:44.920076+0000 2024-03-25T23:13:09.306 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:13:10.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:13:10.308 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:13:10.308 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-25T23:13:10.308 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-25T23:13:10.308 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:13:10.309 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:13:10.309 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-25T23:13:10.625 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:13:05.723285+0000 '>' 2024-03-25T23:12:44.920076+0000 2024-03-25T23:13:10.625 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-25T23:13:10.625 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-25T23:13:10.625 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:13:10.625 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:13:10.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-25T23:13:10.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:13:10.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-25T23:13:10.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:13:10.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:13:10.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-25T23:13:10.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:13:10.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-25T23:13:10.626 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:13:10.626 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:13:10.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:13:10.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:13:10.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:13:10.935 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:13:10.935 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-25T23:13:10.935 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:13:10.935 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:13:10.935 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-25T23:13:10.935 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:13:10.936 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:13:10.936 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-25T23:13:11.238 INFO:tasks.workunit.client.0.smithi129.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-03-25T23:13:11.239 INFO:tasks.workunit.client.0.smithi129.stdout:_ 2024-03-25T23:13:11.239 INFO:tasks.workunit.client.0.smithi129.stdout:snapset 2024-03-25T23:13:11.544 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-25T23:13:11.545 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:13:11.545 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:13:11.545 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-25T23:13:11.545 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:13:11.545 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:13:11.545 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:13:11.545 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:13:11.545 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:13:11.545 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:13:11.545 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-25T23:13:11.545 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-25T23:13:11.545 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:13:11.545 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:13:11.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:13:11.546 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:13:11.546 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:13:11.546 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:13:11.546 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:13:11.546 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:13:11.546 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:13:11.546 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:13:11.548 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:13:11.548 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:13:11.548 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:13:11.548 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:13:11.549 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:13:11.549 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:13:11.549 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:13:11.549 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:13:11.549 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:13:11.549 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:13:11.549 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-25T23:13:11.549 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-25T23:13:11.549 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.1 2024-03-25T23:13:11.549 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:13:11.549 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-25T23:13:11.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-25T23:13:11.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:13:11.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:13:11.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:13:11.581 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:13:11.583+0000 7f23f01e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:13:11.599 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:13:11.599+0000 7f23f01e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:13:11.601 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:13:11.603+0000 7f23f01e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:13:11.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-25T23:13:11.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:13:11.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T23:13:11.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:13:11.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:13:11.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:13:11.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:13:11.861 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-25T23:13:11.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:13:11.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:13:12.138 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:13:12.139+0000 7f23f01e37c0 -1 Falling back to public interface 2024-03-25T23:13:12.184 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:13:13.185 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:13:13.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:13:13.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:13:13.186 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:13:13.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:13:13.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:13:13.495 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:13:14.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:13:14.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:13:14.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:13:14.497 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:13:14.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:13:14.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:13:14.818 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:13:15.142 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:13:15.143+0000 7f23f01e37c0 -1 osd.1 28 log_to_monitors true 2024-03-25T23:13:15.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:13:15.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:13:15.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:13:15.820 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:13:15.821 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:13:15.821 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:13:16.172 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1 up in weight 1 up_from 32 up_thru 32 down_at 29 last_clean_interval [10,28) [v2:127.0.0.1:6810/2418450161,v1:127.0.0.1:6811/2418450161] [v2:127.0.0.1:6812/2418450161,v1:127.0.0.1:6813/2418450161] exists,up 8976cf0a-9b81-4233-8ef6-5c9b582737e8 2024-03-25T23:13:16.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:13:16.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:13:16.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:13:16.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:13:16.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:13:16.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:13:16.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:13:16.173 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:13:16.173 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:13:16.174 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:13:16.174 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:13:16.174 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:13:16.174 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:13:16.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:13:16.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:13:16.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:13:16.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:13:16.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:13:16.281 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:13:16.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:13:16.732 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-25T23:13:16.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:13:16.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:13:16.733 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:13:16.981 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149702 2024-03-25T23:13:16.981 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149702 2024-03-25T23:13:16.982 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149702' 2024-03-25T23:13:16.982 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:13:16.982 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:13:17.187 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953474 2024-03-25T23:13:17.187 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953474 2024-03-25T23:13:17.187 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149702 1-137438953474' 2024-03-25T23:13:17.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:13:17.188 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149702 2024-03-25T23:13:17.188 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:13:17.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:13:17.190 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149702 2024-03-25T23:13:17.190 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:13:17.191 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149702 2024-03-25T23:13:17.191 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149702' 2024-03-25T23:13:17.191 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 111669149702 2024-03-25T23:13:17.192 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:13:17.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149702 -lt 111669149702 2024-03-25T23:13:17.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:13:17.498 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-137438953474 2024-03-25T23:13:17.498 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:13:17.500 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:13:17.500 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-137438953474 2024-03-25T23:13:17.500 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:13:17.502 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953474 2024-03-25T23:13:17.502 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 137438953474' 2024-03-25T23:13:17.502 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 137438953474 2024-03-25T23:13:17.502 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:13:17.824 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953474 -lt 137438953474 2024-03-25T23:13:17.824 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:13:17.825 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:13:17.825 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:13:18.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-25T23:13:18.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:13:18.229 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:13:18.229 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:13:18.230 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:13:18.230 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:13:18.230 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:13:18.230 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:13:18.557 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T23:13:18.558 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:13:18.558 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:13:18.558 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:13:18.921 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-25T23:13:18.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:13:18.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:13:18.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-03-25T23:13:18.955 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-25T23:13:19.007 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-25T23:13:19.007 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-25T23:13:19.007 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T23:13:19.007 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-25T23:13:19.007 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:13:19.007 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:13:19.007 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:13:19.007 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:13:19.007 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:13:19.317 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:13:19.317 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T23:13:19.318 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T23:13:19.319 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T23:13:19.321 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T23:13:19.321 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T23:13:19.321 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T23:13:19.322 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:13:19.322 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T23:13:19.323 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T23:13:19.323 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:13:19.324 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T23:13:19.325 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T23:13:19.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-25T23:13:19.345 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T23:13:19.345 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:13:19.345 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:13:19.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-25T23:13:19.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T23:13:19.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T23:13:19.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-25T23:13:19.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-25T23:13:19.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-25T23:13:19.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-25T23:13:19.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-25T23:13:19.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T23:13:19.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-25T23:13:19.347 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:13:19.347 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:13:19.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:13:19.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:13:19.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:13:19.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:13:19.350 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T23:13:19.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T23:13:19.350 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T23:13:19.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T23:13:19.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T23:13:19.350 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T23:13:19.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:13:19.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T23:13:19.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T23:13:19.352 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:13:19.353 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T23:13:19.354 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T23:13:19.354 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-25T23:13:19.355 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T23:13:19.355 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:13:19.355 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:13:19.356 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-25T23:13:19.357 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T23:13:19.357 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T23:13:19.357 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-25T23:13:19.358 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-25T23:13:19.358 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:13:19.359 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:13:19.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61636 2024-03-25T23:13:19.363 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-25T23:13:19.363 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-25T23:13:19.363 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-25T23:13:19.363 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-25T23:13:19.363 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_basic td/osd-scrub-repair 2024-03-25T23:13:19.364 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:438: TEST_auto_repair_bluestore_basic: local dir=td/osd-scrub-repair 2024-03-25T23:13:19.364 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:439: TEST_auto_repair_bluestore_basic: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='1' ['pool_name']='testpool' ['extras']=' --osd_scrub_auto_repair=true') 2024-03-25T23:13:19.364 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:439: TEST_auto_repair_bluestore_basic: local -A cluster_conf 2024-03-25T23:13:19.364 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:445: TEST_auto_repair_bluestore_basic: local extr_dbg=3 2024-03-25T23:13:19.364 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:446: TEST_auto_repair_bluestore_basic: standard_scrub_cluster td/osd-scrub-repair cluster_conf 2024-03-25T23:13:19.364 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:221: standard_scrub_cluster: local dir=td/osd-scrub-repair 2024-03-25T23:13:19.365 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:222: standard_scrub_cluster: local -n args=cluster_conf 2024-03-25T23:13:19.365 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:224: standard_scrub_cluster: local OSDS=3 2024-03-25T23:13:19.365 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:225: standard_scrub_cluster: local pg_num=1 2024-03-25T23:13:19.365 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:226: standard_scrub_cluster: local poolname=testpool 2024-03-25T23:13:19.365 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:227: standard_scrub_cluster: args['pool_name']=testpool 2024-03-25T23:13:19.365 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:228: standard_scrub_cluster: local 'extra_pars= --osd_scrub_auto_repair=true' 2024-03-25T23:13:19.366 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local debug_msg=dbg 2024-03-25T23:13:19.366 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local saved_echo_flag=x 2024-03-25T23:13:19.366 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: set +x 2024-03-25T23:13:21.166 INFO:tasks.workunit.client.0.smithi129.stdout:add osd0 574ed783-2f5d-4c67-a1e9-cd24176d347c 2024-03-25T23:13:21.445 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:13:21.501 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:13:21.503+0000 7f38690927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:13:21.503 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:13:21.503+0000 7f38690927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:13:21.505 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:13:21.507+0000 7f38690927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:13:21.505 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:13:21.507+0000 7f38690927c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-25T23:13:21.505 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:13:21.507+0000 7f38690927c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-25T23:13:21.505 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:13:21.507+0000 7f38690927c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-25T23:13:21.587 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:13:21.587+0000 7f38690927c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-25T23:13:27.803 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd0 key to auth repository 2024-03-25T23:13:28.178 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.0 2024-03-25T23:13:28.216 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:13:28.215+0000 7fc2f42d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:13:28.244 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:13:28.243+0000 7fc2f42d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:13:28.246 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:13:28.247+0000 7fc2f42d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:13:28.522 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:13:28.802 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:13:28.803+0000 7fc2f42d37c0 -1 Falling back to public interface 2024-03-25T23:13:29.877 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:13:31.183 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:13:31.183+0000 7fc2f42d37c0 -1 osd.0 0 log_to_monitors true 2024-03-25T23:13:31.188 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:13:32.500 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:13:33.823 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:13:35.283 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:13:36.606 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:13:38.067 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:13:39.384 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:13:40.701 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:13:42.041 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:13:43.367 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:13:44.673 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:13:46.335 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:13:47.987 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:13:49.432 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:13:51.075 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:13:52.838 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:13:54.333 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:13:55.922 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:13:56.294 INFO:tasks.workunit.client.0.smithi129.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/807906955,v1:127.0.0.1:6803/807906955] [v2:127.0.0.1:6804/807906955,v1:127.0.0.1:6805/807906955] exists,up 574ed783-2f5d-4c67-a1e9-cd24176d347c 2024-03-25T23:13:56.299 INFO:tasks.workunit.client.0.smithi129.stdout:add osd1 76f90243-5d52-4412-82ea-679873a14349 2024-03-25T23:13:56.694 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:13:56.755 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:13:56.754+0000 7fbc557507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:13:56.795 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:13:56.794+0000 7fbc557507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:13:56.797 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:13:56.798+0000 7fbc557507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:13:56.797 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:13:56.798+0000 7fbc557507c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-25T23:13:56.798 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:13:56.798+0000 7fbc557507c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-25T23:13:56.798 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:13:56.798+0000 7fbc557507c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-25T23:13:56.874 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:13:56.874+0000 7fbc557507c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-25T23:14:03.785 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd1 key to auth repository 2024-03-25T23:14:04.175 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.1 2024-03-25T23:14:04.213 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:14:04.214+0000 7f90bd9707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:14:04.244 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:14:04.242+0000 7f90bd9707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:14:04.246 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:14:04.246+0000 7f90bd9707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:14:04.510 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:14:04.802 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:14:04.802+0000 7f90bd9707c0 -1 Falling back to public interface 2024-03-25T23:14:05.853 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:14:07.167 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:14:07.370 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:14:07.370+0000 7f90bd9707c0 -1 osd.1 0 log_to_monitors true 2024-03-25T23:14:08.493 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:14:09.831 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:14:11.281 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:14:12.592 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:14:14.050 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:14:15.478 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:14:16.779 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:14:18.234 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:14:19.567 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:14:21.023 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:14:22.465 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:14:23.934 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:14:25.543 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:14:27.472 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:14:29.302 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:14:31.199 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:14:32.945 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:14:34.534 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-25T23:14:34.853 INFO:tasks.workunit.client.0.smithi129.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1898973368,v1:127.0.0.1:6811/1898973368] [v2:127.0.0.1:6812/1898973368,v1:127.0.0.1:6813/1898973368] exists,up 76f90243-5d52-4412-82ea-679873a14349 2024-03-25T23:14:34.857 INFO:tasks.workunit.client.0.smithi129.stdout:add osd2 036d1f05-8bb3-49d6-bc51-11ae58e354df 2024-03-25T23:14:35.269 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:14:35.331 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:14:35.334+0000 7f84313c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:14:35.405 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:14:35.410+0000 7f84313c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:14:35.407 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:14:35.410+0000 7f84313c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:14:35.407 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:14:35.410+0000 7f84313c57c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-25T23:14:35.408 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:14:35.410+0000 7f84313c57c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-25T23:14:35.408 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:14:35.410+0000 7f84313c57c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-25T23:14:35.483 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:14:35.486+0000 7f84313c57c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-25T23:14:42.453 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd2 key to auth repository 2024-03-25T23:14:42.826 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.2 2024-03-25T23:14:42.864 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:14:42.865+0000 7f72b7ecc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:14:42.888 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:14:42.889+0000 7f72b7ecc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:14:42.889 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:14:42.893+0000 7f72b7ecc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:14:43.146 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:14:43.438 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:14:43.441+0000 7f72b7ecc7c0 -1 Falling back to public interface 2024-03-25T23:14:44.488 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:14:45.797 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:14:46.132 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:14:46.133+0000 7f72b7ecc7c0 -1 osd.2 0 log_to_monitors true 2024-03-25T23:14:47.117 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:14:48.577 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:14:49.956 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:14:51.322 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:14:52.648 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:14:54.103 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:14:55.553 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:14:56.893 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:14:58.293 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:14:59.694 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:15:01.167 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:15:02.474 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:15:03.882 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:15:05.766 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:15:07.230 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:15:08.945 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:15:10.349 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:15:12.306 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-25T23:15:13.963 INFO:tasks.workunit.client.0.smithi129.stdout:21 2024-03-25T23:15:14.332 INFO:tasks.workunit.client.0.smithi129.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/770531202,v1:127.0.0.1:6819/770531202] [v2:127.0.0.1:6820/770531202,v1:127.0.0.1:6821/770531202] exists,up 036d1f05-8bb3-49d6-bc51-11ae58e354df 2024-03-25T23:15:14.743 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'testpool' created 2024-03-25T23:15:16.835 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 21474836503 2024-03-25T23:15:18.461 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 42949672974 2024-03-25T23:15:18.839 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 64424509448 2024-03-25T23:15:21.598 INFO:tasks.workunit.client.0.smithi129.stdout:standard_scrub_cluster: dbg: test pool is testpool 1 2024-03-25T23:15:21.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:447: TEST_auto_repair_bluestore_basic: local poolid=1 2024-03-25T23:15:21.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:448: TEST_auto_repair_bluestore_basic: local poolname=testpool 2024-03-25T23:15:21.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:450: TEST_auto_repair_bluestore_basic: ceph osd pool set testpool size 2 2024-03-25T23:15:22.059 INFO:tasks.workunit.client.0.smithi129.stderr:set pool 1 size to 2 2024-03-25T23:15:22.079 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:451: TEST_auto_repair_bluestore_basic: wait_for_clean 2024-03-25T23:15:22.079 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:15:22.079 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:15:22.079 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:15:22.079 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:15:22.080 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:15:22.080 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:15:22.080 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:15:22.080 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:15:22.081 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:15:22.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:15:22.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:15:22.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:15:22.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:15:22.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:15:22.195 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:15:22.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:15:22.630 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:15:22.630 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-25T23:15:22.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:15:22.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:15:22.631 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:15:22.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836506 2024-03-25T23:15:22.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836506 2024-03-25T23:15:22.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506' 2024-03-25T23:15:22.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:15:22.777 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:15:22.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672976 2024-03-25T23:15:22.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672976 2024-03-25T23:15:22.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-42949672976' 2024-03-25T23:15:22.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:15:22.919 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:15:23.058 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509450 2024-03-25T23:15:23.058 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509450 2024-03-25T23:15:23.058 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-42949672976 2-64424509450' 2024-03-25T23:15:23.058 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:15:23.058 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836506 2024-03-25T23:15:23.059 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:15:23.059 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:15:23.059 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836506 2024-03-25T23:15:23.060 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:15:23.061 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836506 2024-03-25T23:15:23.061 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 21474836506 2024-03-25T23:15:23.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-03-25T23:15:23.062 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:15:23.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836506 2024-03-25T23:15:23.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:15:24.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:15:24.349 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:15:24.669 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836506 2024-03-25T23:15:24.669 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:15:24.669 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672976 2024-03-25T23:15:24.670 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:15:24.674 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:15:24.674 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672976 2024-03-25T23:15:24.674 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:15:24.675 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672976 2024-03-25T23:15:24.675 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 42949672976 2024-03-25T23:15:24.675 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-03-25T23:15:24.675 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:15:25.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672976 2024-03-25T23:15:25.024 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:15:25.024 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509450 2024-03-25T23:15:25.024 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:15:25.025 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:15:25.027 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509450 2024-03-25T23:15:25.027 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:15:25.028 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509450 2024-03-25T23:15:25.028 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 64424509450 2024-03-25T23:15:25.029 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-03-25T23:15:25.029 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:15:25.408 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509450 -lt 64424509450 2024-03-25T23:15:25.408 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:15:25.409 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:15:25.409 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:15:25.804 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-25T23:15:25.804 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:15:25.804 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:15:25.804 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:15:25.805 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:15:25.805 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:15:25.805 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:15:25.805 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:15:26.098 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-25T23:15:26.102 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:15:26.103 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:15:26.103 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:15:26.476 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-25T23:15:26.477 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:15:26.477 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:15:26.477 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:454: TEST_auto_repair_bluestore_basic: local payload=ABCDEF 2024-03-25T23:15:26.477 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:455: TEST_auto_repair_bluestore_basic: echo ABCDEF 2024-03-25T23:15:26.477 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:456: TEST_auto_repair_bluestore_basic: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-25T23:15:26.655 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-03-25T23:15:26.655 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-25T23:15:26.656 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-25T23:15:26.656 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-25T23:15:26.656 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-25T23:15:26.656 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-25T23:15:26.657 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-25T23:15:26.657 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-25T23:15:26.971 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-25T23:15:26.972 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-25T23:15:26.972 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-25T23:15:27.302 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-03-25T23:15:27.302 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:15:27.302 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:15:27.302 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-25T23:15:27.302 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:15:27.302 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-03-25T23:15:27.302 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:15:27.302 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:15:27.302 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-25T23:15:27.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:15:27.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-25T23:15:27.303 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:15:27.303 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:15:27.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:15:27.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:15:27.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:15:27.611 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:15:27.611 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-03-25T23:15:27.611 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:15:27.611 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:15:27.611 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-25T23:15:27.611 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:15:27.612 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-25T23:15:27.612 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-03-25T23:15:29.112 INFO:tasks.workunit.client.0.smithi129.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-03-25T23:15:29.647 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-25T23:15:29.647 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:15:29.647 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:15:29.647 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-25T23:15:29.647 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:15:29.647 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-25T23:15:29.647 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:15:29.647 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:15:29.647 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:15:29.647 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:15:29.647 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-25T23:15:29.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-25T23:15:29.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:15:29.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:15:29.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:15:29.648 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:15:29.648 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:15:29.648 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:15:29.649 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:15:29.649 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:15:29.649 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:15:29.650 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:15:29.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:15:29.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:15:29.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:15:29.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:15:29.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:15:29.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:15:29.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:15:29.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:15:29.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:15:29.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:15:29.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-25T23:15:29.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-25T23:15:29.651 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.0 2024-03-25T23:15:29.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:15:29.651 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-25T23:15:29.652 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-25T23:15:29.652 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:15:29.652 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:15:29.652 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:15:29.684 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:15:29.684+0000 7f475ceda7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:15:29.687 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:15:29.688+0000 7f475ceda7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:15:29.688 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:15:29.692+0000 7f475ceda7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:15:29.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-25T23:15:29.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:15:29.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T23:15:29.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:15:29.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:15:29.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:15:29.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:15:29.973 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-25T23:15:29.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:15:29.974 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:15:30.223 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:15:30.224+0000 7f475ceda7c0 -1 Falling back to public interface 2024-03-25T23:15:30.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:15:31.320 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:15:31.321 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:15:31.321 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:15:31.321 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:15:31.321 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:15:31.321 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:15:31.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:15:32.647 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:15:32.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:15:32.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:15:32.648 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:15:32.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:15:32.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:15:32.894 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:15:32.896+0000 7f475ceda7c0 -1 osd.0 21 log_to_monitors true 2024-03-25T23:15:32.970 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:15:33.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:15:33.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:15:33.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:15:33.972 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:15:33.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:15:33.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:15:34.307 INFO:tasks.workunit.client.0.smithi129.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/1450932847,v1:127.0.0.1:6803/1450932847] [v2:127.0.0.1:6804/1450932847,v1:127.0.0.1:6805/1450932847] exists,up 574ed783-2f5d-4c67-a1e9-cd24176d347c 2024-03-25T23:15:34.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:15:34.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:15:34.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:15:34.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:15:34.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:15:34.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:15:34.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:15:34.307 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:15:34.308 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:15:34.308 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:15:34.308 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:15:34.308 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:15:34.309 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:15:34.402 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:15:34.402 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:15:34.402 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:15:34.403 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:15:34.403 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:15:34.403 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:15:34.763 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:15:34.763 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:15:34.763 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-25T23:15:34.763 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:15:34.763 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:15:34.763 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:15:34.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-03-25T23:15:34.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-03-25T23:15:34.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-03-25T23:15:34.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:15:34.911 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:15:35.044 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672980 2024-03-25T23:15:35.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672980 2024-03-25T23:15:35.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672980' 2024-03-25T23:15:35.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:15:35.045 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:15:35.185 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509454 2024-03-25T23:15:35.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509454 2024-03-25T23:15:35.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672980 2-64424509454' 2024-03-25T23:15:35.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:15:35.186 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-03-25T23:15:35.186 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:15:35.187 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:15:35.187 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-03-25T23:15:35.188 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:15:35.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-03-25T23:15:35.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-03-25T23:15:35.189 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 107374182402 2024-03-25T23:15:35.189 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:15:35.478 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-03-25T23:15:35.478 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:15:36.480 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:15:36.481 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:15:36.801 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-03-25T23:15:36.801 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:15:36.801 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:15:36.801 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672980 2024-03-25T23:15:36.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:15:36.803 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672980 2024-03-25T23:15:36.804 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:15:36.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672980 2024-03-25T23:15:36.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-03-25T23:15:36.805 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 42949672980 2024-03-25T23:15:36.805 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:15:37.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672980 -lt 42949672980 2024-03-25T23:15:37.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:15:37.139 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509454 2024-03-25T23:15:37.139 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:15:37.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:15:37.140 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509454 2024-03-25T23:15:37.140 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:15:37.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509454 2024-03-25T23:15:37.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-03-25T23:15:37.142 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 64424509454 2024-03-25T23:15:37.142 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:15:37.514 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509454 -lt 64424509454 2024-03-25T23:15:37.515 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:15:37.515 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:15:37.515 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:15:37.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-25T23:15:37.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:15:37.918 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:15:37.918 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:15:37.918 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:15:37.918 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:15:37.918 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:15:37.918 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:15:38.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-25T23:15:38.257 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:15:38.257 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:15:38.257 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:15:38.604 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-25T23:15:38.604 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:15:38.604 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:15:38.604 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:461: TEST_auto_repair_bluestore_basic: ceph tell 'osd.*' config set osd_scrub_auto_repair true 2024-03-25T23:15:38.760 INFO:tasks.workunit.client.0.smithi129.stdout:osd.0: { 2024-03-25T23:15:38.761 INFO:tasks.workunit.client.0.smithi129.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-03-25T23:15:38.761 INFO:tasks.workunit.client.0.smithi129.stdout:} 2024-03-25T23:15:38.774 INFO:tasks.workunit.client.0.smithi129.stdout:osd.1: { 2024-03-25T23:15:38.774 INFO:tasks.workunit.client.0.smithi129.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-03-25T23:15:38.774 INFO:tasks.workunit.client.0.smithi129.stdout:} 2024-03-25T23:15:38.787 INFO:tasks.workunit.client.0.smithi129.stdout:osd.2: { 2024-03-25T23:15:38.787 INFO:tasks.workunit.client.0.smithi129.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-03-25T23:15:38.787 INFO:tasks.workunit.client.0.smithi129.stdout:} 2024-03-25T23:15:38.802 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:463: TEST_auto_repair_bluestore_basic: get_pg testpool SOMETHING 2024-03-25T23:15:38.802 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-03-25T23:15:38.802 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-25T23:15:38.803 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool SOMETHING 2024-03-25T23:15:38.803 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-25T23:15:39.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:463: TEST_auto_repair_bluestore_basic: local pgid=1.0 2024-03-25T23:15:39.096 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:464: TEST_auto_repair_bluestore_basic: get_primary testpool SOMETHING 2024-03-25T23:15:39.096 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-25T23:15:39.096 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-25T23:15:39.096 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-25T23:15:39.096 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-25T23:15:39.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:464: TEST_auto_repair_bluestore_basic: local primary=1 2024-03-25T23:15:39.419 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:465: TEST_auto_repair_bluestore_basic: get_last_scrub_stamp 1.0 2024-03-25T23:15:39.419 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-25T23:15:39.419 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:15:39.419 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:15:39.419 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-25T23:15:39.728 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:465: TEST_auto_repair_bluestore_basic: local last_scrub_stamp=2024-03-25T23:15:14.697405+0000 2024-03-25T23:15:39.728 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:468: TEST_auto_repair_bluestore_basic: ceph tell 1.0 schedule-deep-scrub 2024-03-25T23:15:39.855 INFO:tasks.workunit.client.0.smithi129.stdout:{ 2024-03-25T23:15:39.855 INFO:tasks.workunit.client.0.smithi129.stdout: "deep": true, 2024-03-25T23:15:39.855 INFO:tasks.workunit.client.0.smithi129.stdout: "must": false, 2024-03-25T23:15:39.855 INFO:tasks.workunit.client.0.smithi129.stdout: "stamp": "2024-03-11T23:13:59.860445+0000" 2024-03-25T23:15:39.855 INFO:tasks.workunit.client.0.smithi129.stdout:} 2024-03-25T23:15:39.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:471: TEST_auto_repair_bluestore_basic: wait_for_scrub 1.0 2024-03-25T23:15:14.697405+0000 2024-03-25T23:15:39.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-03-25T23:15:39.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-25T23:15:14.697405+0000 2024-03-25T23:15:39.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-25T23:15:39.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-25T23:15:39.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:15:39.869 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-25T23:15:39.869 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-25T23:15:39.869 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:15:39.870 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:15:39.870 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-25T23:15:40.148 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:15:14.697405+0000 '>' 2024-03-25T23:15:14.697405+0000 2024-03-25T23:15:40.148 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:15:41.150 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:15:41.150 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:15:41.151 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-25T23:15:41.152 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-25T23:15:41.152 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:15:41.152 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:15:41.152 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-25T23:15:41.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:15:14.697405+0000 '>' 2024-03-25T23:15:14.697405+0000 2024-03-25T23:15:41.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:15:42.462 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:15:42.462 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:15:42.463 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-25T23:15:42.463 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-25T23:15:42.463 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:15:42.464 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:15:42.464 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-25T23:15:42.772 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-11T23:13:59.860445+0000 '>' 2024-03-25T23:15:14.697405+0000 2024-03-25T23:15:42.772 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:15:43.773 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:15:43.773 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:15:43.773 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-25T23:15:43.774 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-25T23:15:43.774 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:15:43.774 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:15:43.774 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-25T23:15:44.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-11T23:13:59.860445+0000 '>' 2024-03-25T23:15:14.697405+0000 2024-03-25T23:15:44.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:15:45.079 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:15:45.080 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:15:45.080 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-25T23:15:45.080 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-25T23:15:45.080 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:15:45.081 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:15:45.081 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-25T23:15:45.380 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-11T23:13:59.860445+0000 '>' 2024-03-25T23:15:14.697405+0000 2024-03-25T23:15:45.380 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:15:46.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:15:46.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:15:46.383 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-25T23:15:46.383 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-25T23:15:46.383 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:15:46.383 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:15:46.384 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-25T23:15:46.691 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:15:40.441970+0000 '>' 2024-03-25T23:15:14.697405+0000 2024-03-25T23:15:46.691 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-25T23:15:46.691 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:472: TEST_auto_repair_bluestore_basic: wait_for_clean 2024-03-25T23:15:46.691 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:15:46.692 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:15:46.692 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:15:46.692 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:15:46.692 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:15:46.692 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:15:46.693 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:15:46.693 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:15:46.693 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:15:46.792 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:15:46.792 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:15:46.792 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:15:46.792 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:15:46.792 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:15:46.792 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:15:47.089 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:15:47.090 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:15:47.090 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-25T23:15:47.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:15:47.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:15:47.090 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:15:47.250 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182405 2024-03-25T23:15:47.250 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182405 2024-03-25T23:15:47.250 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405' 2024-03-25T23:15:47.250 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:15:47.250 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:15:47.399 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672983 2024-03-25T23:15:47.401 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672983 2024-03-25T23:15:47.401 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405 1-42949672983' 2024-03-25T23:15:47.401 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:15:47.401 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:15:47.529 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509457 2024-03-25T23:15:47.529 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509457 2024-03-25T23:15:47.529 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405 1-42949672983 2-64424509457' 2024-03-25T23:15:47.529 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:15:47.530 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182405 2024-03-25T23:15:47.530 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:15:47.530 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:15:47.531 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182405 2024-03-25T23:15:47.531 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:15:47.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182405 2024-03-25T23:15:47.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182405' 2024-03-25T23:15:47.532 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 107374182405 2024-03-25T23:15:47.533 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:15:47.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182405 -lt 107374182405 2024-03-25T23:15:47.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:15:47.839 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672983 2024-03-25T23:15:47.839 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:15:47.841 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:15:47.842 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672983 2024-03-25T23:15:47.842 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:15:47.843 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672983 2024-03-25T23:15:47.843 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-03-25T23:15:47.843 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 42949672983 2024-03-25T23:15:47.843 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:15:48.128 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672983 -lt 42949672983 2024-03-25T23:15:48.128 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:15:48.129 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509457 2024-03-25T23:15:48.129 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:15:48.131 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:15:48.131 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509457 2024-03-25T23:15:48.131 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:15:48.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509457 2024-03-25T23:15:48.133 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 64424509457 2024-03-25T23:15:48.133 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-03-25T23:15:48.133 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:15:48.469 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509457 -lt 64424509457 2024-03-25T23:15:48.469 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:15:48.470 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:15:48.470 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:15:48.864 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-25T23:15:48.865 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:15:48.866 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:15:48.866 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:15:48.866 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:15:48.866 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:15:48.866 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:15:48.866 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:15:49.203 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-25T23:15:49.203 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:15:49.204 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:15:49.204 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:15:49.556 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-25T23:15:49.556 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:15:49.556 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:15:49.556 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:473: TEST_auto_repair_bluestore_basic: ceph pg dump pgs 2024-03-25T23:15:49.842 INFO:tasks.workunit.client.0.smithi129.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-25T23:15:49.842 INFO:tasks.workunit.client.0.smithi129.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-03-25T23:15:40.665817+0000 21'1 26:57 [1,0] 1 [1,0] 1 21'1 2024-03-25T23:15:40.441970+0000 21'1 2024-03-25T23:15:40.441970+0000 0 0 periodic scrub scheduled @ 2024-03-26T23:15:40.441970+0000 1 0 2024-03-25T23:15:49.842 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:15:49.842 INFO:tasks.workunit.client.0.smithi129.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-25T23:15:49.842 INFO:tasks.workunit.client.0.smithi129.stderr:dumped pgs 2024-03-25T23:15:49.859 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:476: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-03-25T23:15:49.859 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-25T23:15:49.859 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-25T23:15:49.859 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-25T23:15:49.860 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-25T23:15:49.860 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-25T23:15:49.860 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-25T23:15:49.860 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-25T23:15:50.160 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-25T23:15:50.160 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-25T23:15:50.161 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-25T23:15:50.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:476: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-25T23:15:50.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:15:50.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:15:50.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-25T23:15:50.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:15:50.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-25T23:15:50.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:15:50.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:15:50.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-25T23:15:50.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:15:50.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-25T23:15:50.494 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:15:50.494 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:15:50.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:15:50.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:15:50.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:15:51.804 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:15:51.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-25T23:15:51.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:15:51.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:15:51.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-25T23:15:51.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:15:51.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-25T23:15:51.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-03-25T23:15:51.904 INFO:tasks.workunit.client.0.smithi129.stdout:_ 2024-03-25T23:15:51.904 INFO:tasks.workunit.client.0.smithi129.stdout:snapset 2024-03-25T23:15:52.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-25T23:15:52.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:15:52.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:15:52.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-25T23:15:52.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:15:52.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-25T23:15:52.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:15:52.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:15:52.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:15:52.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:15:52.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-25T23:15:52.208 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-25T23:15:52.208 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:15:52.208 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:15:52.208 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:15:52.208 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:15:52.208 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:15:52.208 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:15:52.208 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:15:52.208 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:15:52.208 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:15:52.209 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:15:52.209 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:15:52.209 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:15:52.209 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:15:52.209 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:15:52.210 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:15:52.210 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:15:52.210 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:15:52.210 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:15:52.210 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:15:52.210 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:15:52.210 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-25T23:15:52.210 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-25T23:15:52.210 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.0 2024-03-25T23:15:52.211 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:15:52.211 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-25T23:15:52.212 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-25T23:15:52.213 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:15:52.213 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:15:52.213 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:15:52.243 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:15:52.244+0000 7fb08a28f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:15:52.267 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:15:52.268+0000 7fb08a28f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:15:52.269 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:15:52.272+0000 7fb08a28f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:15:52.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-25T23:15:52.563 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:15:52.563 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T23:15:52.563 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:15:52.563 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:15:52.563 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:15:52.563 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:15:52.563 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-25T23:15:52.563 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:15:52.563 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:15:52.810 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:15:52.812+0000 7fb08a28f7c0 -1 Falling back to public interface 2024-03-25T23:15:52.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:15:53.901 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:15:53.901 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:15:53.901 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:15:53.901 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:15:53.902 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:15:53.902 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:15:54.216 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:15:55.218 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:15:55.218 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:15:55.218 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:15:55.218 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:15:55.218 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:15:55.219 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:15:55.537 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:15:55.883 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:15:55.884+0000 7fb08a28f7c0 -1 osd.0 26 log_to_monitors true 2024-03-25T23:15:56.538 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:15:56.539 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:15:56.539 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:15:56.539 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:15:56.539 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:15:56.539 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:15:56.857 INFO:tasks.workunit.client.0.smithi129.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/2767609492,v1:127.0.0.1:6803/2767609492] [v2:127.0.0.1:6804/2767609492,v1:127.0.0.1:6805/2767609492] exists,up 574ed783-2f5d-4c67-a1e9-cd24176d347c 2024-03-25T23:15:56.857 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:15:56.857 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:15:56.857 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:15:56.858 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:15:56.858 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:15:56.858 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:15:56.858 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:15:56.858 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:15:56.859 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:15:56.859 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:15:56.859 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:15:56.859 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:15:56.859 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:15:56.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:15:56.957 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:15:56.957 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:15:56.957 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:15:56.957 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:15:56.957 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:15:57.269 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:15:57.269 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:15:57.269 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-25T23:15:57.269 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:15:57.269 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:15:57.270 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:15:57.417 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-03-25T23:15:57.417 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-03-25T23:15:57.417 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882' 2024-03-25T23:15:57.417 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:15:57.417 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:15:57.573 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672986 2024-03-25T23:15:57.573 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672986 2024-03-25T23:15:57.574 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672986' 2024-03-25T23:15:57.574 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:15:57.574 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:15:57.754 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509460 2024-03-25T23:15:57.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509460 2024-03-25T23:15:57.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672986 2-64424509460' 2024-03-25T23:15:57.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:15:57.755 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018882 2024-03-25T23:15:57.755 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:15:57.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:15:57.757 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018882 2024-03-25T23:15:57.757 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:15:57.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-03-25T23:15:57.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-03-25T23:15:57.758 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 128849018882 2024-03-25T23:15:57.758 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:15:58.150 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-03-25T23:15:58.150 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:15:58.151 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672986 2024-03-25T23:15:58.151 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:15:58.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:15:58.153 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672986 2024-03-25T23:15:58.154 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:15:58.155 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672986 2024-03-25T23:15:58.155 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2024-03-25T23:15:58.155 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 42949672986 2024-03-25T23:15:58.155 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:15:58.463 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672986 -lt 42949672986 2024-03-25T23:15:58.463 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:15:58.464 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509460 2024-03-25T23:15:58.464 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:15:58.465 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:15:58.466 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509460 2024-03-25T23:15:58.466 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:15:58.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509460 2024-03-25T23:15:58.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-03-25T23:15:58.468 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 64424509460 2024-03-25T23:15:58.468 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:15:58.804 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509459 -lt 64424509460 2024-03-25T23:15:58.804 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:15:59.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:15:59.806 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:16:00.128 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509461 -lt 64424509460 2024-03-25T23:16:00.129 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:16:00.129 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:16:00.129 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:16:00.521 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-25T23:16:00.521 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:16:00.521 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:16:00.521 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:16:00.521 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:16:00.522 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:16:00.522 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:16:00.522 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:16:00.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-25T23:16:00.862 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:16:00.862 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:16:00.863 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:16:01.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-03-25T23:16:01.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-03-25T23:16:01.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-25T23:16:01.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-03-25T23:16:01.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:16:01.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-25T23:16:01.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:16:01.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:16:01.331 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:16:01.331 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:16:01.331 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:16:01.331 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:16:01.331 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:16:01.331 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:16:01.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-25T23:16:01.627 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:16:01.628 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:16:01.628 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:16:02.004 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-03-25T23:16:02.004 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-03-25T23:16:02.004 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-25T23:16:02.004 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-25T23:16:02.004 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-25T23:16:02.004 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-25T23:16:02.004 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-25T23:16:02.005 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-25T23:16:02.005 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-25T23:16:02.397 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-25T23:16:02.397 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-25T23:16:02.397 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-25T23:16:02.397 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:16:02.397 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-25T23:16:02.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:16:02.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:16:02.599 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:16:02.599 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:16:02.599 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:16:02.599 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:16:02.599 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:16:02.599 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:16:02.890 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-25T23:16:02.891 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:16:02.891 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:16:02.891 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:16:03.245 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-25T23:16:03.245 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:16:03.245 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:16:03.245 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:477: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-03-25T23:16:03.245 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-25T23:16:03.245 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-25T23:16:03.246 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-25T23:16:03.246 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-25T23:16:03.246 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-25T23:16:03.246 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-25T23:16:03.246 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-25T23:16:03.548 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-25T23:16:03.549 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-25T23:16:03.549 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-25T23:16:03.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:477: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-03-25T23:16:03.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:16:03.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:16:03.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-25T23:16:03.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:16:03.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-03-25T23:16:03.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:16:03.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:16:03.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-25T23:16:03.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:16:03.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-25T23:16:03.923 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:16:03.924 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:16:03.924 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:16:03.924 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:16:03.924 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:16:04.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:16:04.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-03-25T23:16:04.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:16:04.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:16:04.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-25T23:16:04.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:16:04.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-25T23:16:04.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-03-25T23:16:04.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-25T23:16:04.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:16:04.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:16:04.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-25T23:16:04.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:16:04.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-25T23:16:04.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:16:04.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:16:04.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:16:04.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:16:04.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-25T23:16:04.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-25T23:16:04.705 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:16:04.705 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:16:04.705 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:16:04.705 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:16:04.705 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:16:04.705 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:16:04.707 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:16:04.707 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:16:04.707 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:16:04.707 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:16:04.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:16:04.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:16:04.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:16:04.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:16:04.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:16:04.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:16:04.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:16:04.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:16:04.708 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:16:04.708 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:16:04.708 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-25T23:16:04.708 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-25T23:16:04.709 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.0 2024-03-25T23:16:04.709 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:16:04.710 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-25T23:16:04.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-25T23:16:04.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:16:04.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:16:04.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:16:04.742 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:16:04.744+0000 7ff5367257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:16:04.779 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:16:04.780+0000 7ff5367257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:16:04.781 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:16:04.784+0000 7ff5367257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:16:05.039 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-25T23:16:05.039 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:16:05.039 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T23:16:05.039 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:16:05.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:16:05.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:16:05.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:16:05.040 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-25T23:16:05.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:16:05.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:16:05.334 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:16:05.336+0000 7ff5367257c0 -1 Falling back to public interface 2024-03-25T23:16:05.405 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:16:06.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:16:06.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:16:06.407 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:16:06.407 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:16:06.407 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:16:06.407 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:16:06.721 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:16:07.723 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:16:07.723 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:16:07.723 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:16:07.723 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:16:07.723 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:16:07.723 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:16:08.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:16:08.429 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:16:08.432+0000 7ff5367257c0 -1 osd.0 31 log_to_monitors true 2024-03-25T23:16:09.044 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:16:09.044 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:16:09.044 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:16:09.044 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:16:09.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:16:09.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:16:09.382 INFO:tasks.workunit.client.0.smithi129.stderr:osd.0 up in weight 1 up_from 35 up_thru 0 down_at 32 last_clean_interval [30,31) [v2:127.0.0.1:6802/366520655,v1:127.0.0.1:6803/366520655] [v2:127.0.0.1:6804/366520655,v1:127.0.0.1:6805/366520655] exists,up 574ed783-2f5d-4c67-a1e9-cd24176d347c 2024-03-25T23:16:09.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:16:09.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:16:09.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:16:09.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:16:09.383 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:16:09.383 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:16:09.383 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:16:09.383 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:16:09.383 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:16:09.384 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:16:09.384 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:16:09.384 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:16:09.384 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:16:09.492 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:16:09.492 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:16:09.492 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:16:09.492 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:16:09.492 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:16:09.492 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:16:09.827 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:16:09.827 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:16:09.827 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-25T23:16:09.827 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:16:09.827 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:16:09.828 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:16:09.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-03-25T23:16:09.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-03-25T23:16:09.980 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362' 2024-03-25T23:16:09.980 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:16:09.980 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:16:10.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672990 2024-03-25T23:16:10.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672990 2024-03-25T23:16:10.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-42949672990' 2024-03-25T23:16:10.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:16:10.182 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:16:10.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509464 2024-03-25T23:16:10.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509464 2024-03-25T23:16:10.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-42949672990 2-64424509464' 2024-03-25T23:16:10.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:16:10.330 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:16:10.330 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855362 2024-03-25T23:16:10.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:16:10.331 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855362 2024-03-25T23:16:10.332 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:16:10.333 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-03-25T23:16:10.333 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2024-03-25T23:16:10.333 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 150323855362 2024-03-25T23:16:10.333 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:16:10.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 150323855362 2024-03-25T23:16:10.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:16:11.625 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:16:11.625 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:16:11.983 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-03-25T23:16:11.983 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:16:11.983 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672990 2024-03-25T23:16:11.984 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:16:11.984 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:16:11.985 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672990 2024-03-25T23:16:11.985 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:16:11.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672990 2024-03-25T23:16:11.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672990' 2024-03-25T23:16:11.987 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 42949672990 2024-03-25T23:16:11.988 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:16:12.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672990 -lt 42949672990 2024-03-25T23:16:12.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:16:12.300 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509464 2024-03-25T23:16:12.300 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:16:12.301 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:16:12.302 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:16:12.302 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509464 2024-03-25T23:16:12.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509464 2024-03-25T23:16:12.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-03-25T23:16:12.303 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 64424509464 2024-03-25T23:16:12.304 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:16:12.622 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509464 -lt 64424509464 2024-03-25T23:16:12.623 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:16:12.624 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:16:12.624 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:16:13.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-25T23:16:13.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:16:13.124 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:16:13.124 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:16:13.124 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:16:13.124 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:16:13.124 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:16:13.124 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:16:13.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-25T23:16:13.453 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:16:13.453 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:16:13.453 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:16:13.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-25T23:16:13.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:16:13.821 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:16:13.821 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:478: TEST_auto_repair_bluestore_basic: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-25T23:16:13.821 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:479: TEST_auto_repair_bluestore_basic: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-03-25T23:16:13.826 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:15:40.436+0000 7f90a18fa640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-03-25T23:16:13.826 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:15:40.436+0000 7f90a18fa640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-03-25T23:16:13.826 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:15:40.436+0000 7f90a18fa640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-03-25T23:16:13.826 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:15:40.436+0000 7f90a18fa640 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-03-25T23:16:13.826 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:15:40.436+0000 7f90a18fa640 19 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-25T23:16:13.827 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-25T23:16:13.827 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-25T23:16:13.827 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T23:16:13.827 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-25T23:16:13.827 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:16:13.828 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:16:13.828 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:16:13.828 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:16:13.828 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:16:14.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:16:14.136 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T23:16:14.137 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T23:16:14.137 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T23:16:14.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T23:16:14.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T23:16:14.140 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T23:16:14.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:16:14.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T23:16:14.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T23:16:14.143 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:16:14.144 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T23:16:14.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T23:16:14.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-25T23:16:14.166 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T23:16:14.166 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:16:14.166 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:16:14.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-25T23:16:14.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T23:16:14.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T23:16:14.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-25T23:16:14.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-25T23:16:14.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-25T23:16:14.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-25T23:16:14.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-25T23:16:14.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T23:16:14.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-25T23:16:14.169 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:16:14.170 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:16:14.170 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:16:14.170 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:16:14.170 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:16:14.171 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:16:14.172 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T23:16:14.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T23:16:14.173 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T23:16:14.174 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T23:16:14.174 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T23:16:14.175 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T23:16:14.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:16:14.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T23:16:14.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T23:16:14.177 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:16:14.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T23:16:14.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T23:16:14.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-25T23:16:14.180 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T23:16:14.180 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:16:14.180 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:16:14.180 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-25T23:16:14.181 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T23:16:14.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T23:16:14.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-25T23:16:14.183 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-25T23:16:14.183 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:16:14.183 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:16:14.183 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61636 2024-03-25T23:16:14.185 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-25T23:16:14.185 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-25T23:16:14.185 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-25T23:16:14.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-25T23:16:14.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_failed td/osd-scrub-repair 2024-03-25T23:16:14.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:539: TEST_auto_repair_bluestore_failed: local dir=td/osd-scrub-repair 2024-03-25T23:16:14.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:540: TEST_auto_repair_bluestore_failed: local poolname=testpool 2024-03-25T23:16:14.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:543: TEST_auto_repair_bluestore_failed: run_mon td/osd-scrub-repair a 2024-03-25T23:16:14.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-25T23:16:14.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-25T23:16:14.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-25T23:16:14.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-25T23:16:14.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-25T23:16:14.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-25T23:16:14.661 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-25T23:16:14.662 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:16:14.662 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:16:14.662 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:16:14.662 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:16:14.662 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:16:14.663 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:16:14.663 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-25T23:16:14.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-25T23:16:14.888 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-25T23:16:14.888 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T23:16:14.888 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T23:16:14.888 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-25T23:16:14.890 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-25T23:16:14.890 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T23:16:14.890 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T23:16:14.890 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T23:16:14.890 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T23:16:14.890 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:16:14.890 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:16:14.891 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-25T23:16:14.891 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T23:16:14.891 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get fsid 2024-03-25T23:16:15.006 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-25T23:16:15.007 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T23:16:15.007 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T23:16:15.007 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-25T23:16:15.008 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T23:16:15.008 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-25T23:16:15.008 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T23:16:15.008 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T23:16:15.008 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T23:16:15.008 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:16:15.008 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:16:15.008 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-25T23:16:15.008 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T23:16:15.009 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get mon_host 2024-03-25T23:16:15.121 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:544: TEST_auto_repair_bluestore_failed: run_mgr td/osd-scrub-repair x 2024-03-25T23:16:15.122 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-25T23:16:15.122 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-25T23:16:15.122 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-25T23:16:15.122 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-25T23:16:15.122 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-25T23:16:15.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-25T23:16:16.004 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-25T23:16:16.005 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:16:16.005 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:16:16.005 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:16:16.006 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:16:16.006 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:16:16.006 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:16:16.006 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T23:16:16.007 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T23:16:16.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:547: TEST_auto_repair_bluestore_failed: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-25T23:16:16.046 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: seq 0 2 2024-03-25T23:16:16.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-03-25T23:16:16.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:16:16.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:16:16.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:16:16.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-25T23:16:16.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:16:16.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-25T23:16:16.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:16:16.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:16:16.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:16:16.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:16:16.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-25T23:16:16.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-25T23:16:16.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:16:16.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:16:16.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:16:16.048 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:16:16.048 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:16:16.048 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:16:16.048 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:16:16.048 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:16:16.048 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:16:16.048 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:16:16.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:16:16.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:16:16.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:16:16.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:16:16.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:16:16.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:16:16.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:16:16.049 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:16:16.049 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:16:16.049 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:16:16.049 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:16:16.049 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-25T23:16:16.049 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-25T23:16:16.049 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:16:16.049 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f73ae165-97fa-4ffd-8fc9-39fa3de80b0b 2024-03-25T23:16:16.049 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f73ae165-97fa-4ffd-8fc9-39fa3de80b0b' 2024-03-25T23:16:16.049 INFO:tasks.workunit.client.0.smithi129.stdout:add osd0 f73ae165-97fa-4ffd-8fc9-39fa3de80b0b 2024-03-25T23:16:16.050 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:16:16.064 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDABQJmpnMTBBAA9oLi4TFJccOYO231q63+0w== 2024-03-25T23:16:16.064 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDABQJmpnMTBBAA9oLi4TFJccOYO231q63+0w=="}' 2024-03-25T23:16:16.064 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f73ae165-97fa-4ffd-8fc9-39fa3de80b0b -i td/osd-scrub-repair/0/new.json 2024-03-25T23:16:16.316 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:16:16.333 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-25T23:16:16.334 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDABQJmpnMTBBAA9oLi4TFJccOYO231q63+0w== --osd-uuid f73ae165-97fa-4ffd-8fc9-39fa3de80b0b 2024-03-25T23:16:16.367 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:16:16.368+0000 7fc70a1747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:16:16.383 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:16:16.384+0000 7fc70a1747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:16:16.384 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:16:16.388+0000 7fc70a1747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:16:16.385 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:16:16.388+0000 7fc70a1747c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-25T23:16:16.385 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:16:16.388+0000 7fc70a1747c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-25T23:16:16.385 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:16:16.388+0000 7fc70a1747c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-25T23:16:16.483 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:16:16.484+0000 7fc70a1747c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-25T23:16:22.794 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-25T23:16:22.794 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:16:22.795 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd0 key to auth repository 2024-03-25T23:16:22.795 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-25T23:16:22.795 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:16:23.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-25T23:16:23.168 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.0 2024-03-25T23:16:23.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:16:23.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:16:23.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:16:23.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:16:23.202 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:16:23.204+0000 7f3b2436d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:16:23.232 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:16:23.232+0000 7f3b2436d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:16:23.233 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:16:23.236+0000 7f3b2436d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:16:23.491 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-25T23:16:23.491 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:16:23.491 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T23:16:23.492 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:16:23.492 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:16:23.492 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:16:23.492 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:16:23.492 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:16:23.492 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:16:23.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:16:23.782 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:16:23.784+0000 7f3b2436d7c0 -1 Falling back to public interface 2024-03-25T23:16:23.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:16:24.822 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:16:24.822 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:16:24.822 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:16:24.823 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:16:24.823 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:16:24.823 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:16:25.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:16:26.144 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:16:26.144 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:16:26.144 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:16:26.144 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:16:26.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:16:26.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:16:26.407 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:16:26.407+0000 7f3b2436d7c0 -1 osd.0 0 log_to_monitors true 2024-03-25T23:16:26.455 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:16:27.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:16:27.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:16:27.457 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:16:27.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:16:27.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:16:27.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:16:27.766 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:16:28.768 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:16:28.768 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:16:28.768 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:16:28.769 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:16:28.769 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:16:28.769 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:16:29.148 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:16:30.150 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:16:30.150 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:16:30.150 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:16:30.151 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:16:30.151 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:16:30.151 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:16:30.486 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:16:31.487 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:16:31.488 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:16:31.488 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:16:31.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:16:31.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:16:31.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:16:31.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:16:32.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:16:32.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:16:32.805 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:16:32.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:16:32.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:16:32.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:16:33.130 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:16:34.131 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:16:34.131 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:16:34.132 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:16:34.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:16:34.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:16:34.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:16:34.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:16:35.501 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:16:35.501 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:16:35.502 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:16:35.502 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:16:35.502 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:16:35.502 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:16:35.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:16:36.822 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:16:36.822 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:16:36.822 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:16:36.823 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:16:36.823 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:16:36.823 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:16:37.137 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:16:38.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:16:38.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:16:38.140 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:16:38.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:16:38.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:16:38.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:16:38.679 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:16:39.681 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:16:39.681 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:16:39.681 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:16:39.682 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:16:39.682 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:16:39.682 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:16:39.997 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:16:40.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:16:40.999 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:16:40.999 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:16:40.999 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:16:40.999 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:16:40.999 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:16:41.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:16:42.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:16:42.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:16:42.862 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:16:42.862 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:16:42.862 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:16:42.862 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:16:43.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:16:44.700 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:16:44.700 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:16:44.700 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:16:44.701 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:16:44.701 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:16:44.701 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:16:45.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:16:46.495 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:16:46.495 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:16:46.496 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:16:46.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:16:46.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:16:46.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:16:47.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:16:48.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:16:48.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:16:48.332 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:16:48.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:16:48.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:16:48.333 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:16:49.164 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:16:50.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:16:50.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:16:50.166 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:16:50.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:16:50.167 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:16:50.167 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:16:50.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:16:51.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:16:51.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:16:51.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:16:51.706 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:16:51.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:16:51.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:16:52.029 INFO:tasks.workunit.client.0.smithi129.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1291209624,v1:127.0.0.1:6803/1291209624] [v2:127.0.0.1:6804/1291209624,v1:127.0.0.1:6805/1291209624] exists,up f73ae165-97fa-4ffd-8fc9-39fa3de80b0b 2024-03-25T23:16:52.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:16:52.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:16:52.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:16:52.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-03-25T23:16:52.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:16:52.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:16:52.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:16:52.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-25T23:16:52.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:16:52.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:16:52.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:16:52.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:16:52.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:16:52.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:16:52.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-25T23:16:52.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-25T23:16:52.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:16:52.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:16:52.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:16:52.034 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:16:52.034 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:16:52.035 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:16:52.035 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:16:52.035 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:16:52.035 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:16:52.035 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:16:52.035 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:16:52.035 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:16:52.035 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:16:52.035 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:16:52.035 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:16:52.035 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:16:52.035 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:16:52.035 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:16:52.035 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:16:52.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:16:52.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:16:52.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-25T23:16:52.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-25T23:16:52.036 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:16:52.036 INFO:tasks.workunit.client.0.smithi129.stdout:add osd1 4993326f-c2cb-4650-a57e-fbe3f3c536c4 2024-03-25T23:16:52.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4993326f-c2cb-4650-a57e-fbe3f3c536c4 2024-03-25T23:16:52.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 4993326f-c2cb-4650-a57e-fbe3f3c536c4' 2024-03-25T23:16:52.037 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:16:52.059 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDkBQJmgqnPAxAAe6Ogdvv2Wnd1EJkY0foZmQ== 2024-03-25T23:16:52.059 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDkBQJmgqnPAxAAe6Ogdvv2Wnd1EJkY0foZmQ=="}' 2024-03-25T23:16:52.060 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4993326f-c2cb-4650-a57e-fbe3f3c536c4 -i td/osd-scrub-repair/1/new.json 2024-03-25T23:16:52.431 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:16:52.449 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-25T23:16:52.450 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDkBQJmgqnPAxAAe6Ogdvv2Wnd1EJkY0foZmQ== --osd-uuid 4993326f-c2cb-4650-a57e-fbe3f3c536c4 2024-03-25T23:16:52.488 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:16:52.487+0000 7fea77b617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:16:52.534 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:16:52.535+0000 7fea77b617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:16:52.535 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:16:52.535+0000 7fea77b617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:16:52.536 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:16:52.539+0000 7fea77b617c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-25T23:16:52.536 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:16:52.539+0000 7fea77b617c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-25T23:16:52.536 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:16:52.539+0000 7fea77b617c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-25T23:16:52.608 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:16:52.607+0000 7fea77b617c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-25T23:16:59.249 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-25T23:16:59.249 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:16:59.251 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd1 key to auth repository 2024-03-25T23:16:59.251 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-25T23:16:59.251 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:16:59.645 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.1 2024-03-25T23:16:59.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-25T23:16:59.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:16:59.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:16:59.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:16:59.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:16:59.681 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:16:59.683+0000 7f60909ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:16:59.712 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:16:59.711+0000 7f60909ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:16:59.713 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:16:59.715+0000 7f60909ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:16:59.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-25T23:16:59.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:16:59.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T23:16:59.963 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:16:59.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:16:59.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:16:59.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:16:59.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:16:59.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:16:59.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:17:00.262 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:17:00.263+0000 7f60909ed7c0 -1 Falling back to public interface 2024-03-25T23:17:00.294 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:17:01.296 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:17:01.296 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:17:01.296 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:17:01.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:17:01.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:17:01.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:17:01.605 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:17:02.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:17:02.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:17:02.607 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:17:02.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:17:02.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:17:02.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:17:02.869 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:17:02.871+0000 7f60909ed7c0 -1 osd.1 0 log_to_monitors true 2024-03-25T23:17:02.925 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:17:03.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:17:03.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:17:03.927 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:17:03.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:17:03.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:17:03.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:17:04.250 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:17:05.252 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:17:05.252 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:17:05.253 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:17:05.253 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:17:05.253 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:17:05.253 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:17:05.574 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:17:06.576 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:17:06.576 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:17:06.576 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:17:06.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:17:06.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:17:06.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:17:07.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:17:08.167 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:17:08.167 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:17:08.168 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:17:08.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:17:08.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:17:08.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:17:08.521 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:17:09.524 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:17:09.524 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:17:09.524 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:17:09.525 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:17:09.525 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:17:09.525 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:17:09.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:17:10.862 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:17:10.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:17:10.863 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:17:10.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:17:10.864 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:17:10.864 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:17:11.187 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:17:12.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:17:12.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:17:12.190 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:17:12.190 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:17:12.190 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:17:12.190 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:17:12.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:17:13.501 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:17:13.501 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:17:13.501 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:17:13.501 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:17:13.502 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:17:13.502 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:17:13.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:17:14.953 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:17:14.953 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:17:14.954 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:17:14.954 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:17:14.954 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:17:14.954 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:17:15.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:17:16.423 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:17:16.423 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:17:16.423 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:17:16.423 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:17:16.423 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:17:16.424 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:17:16.845 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:17:17.847 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:17:17.847 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:17:17.847 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:17:17.848 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:17:17.848 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:17:17.848 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:17:18.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:17:19.375 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:17:19.376 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:17:19.376 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:17:19.377 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:17:19.377 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:17:19.377 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:17:20.089 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:17:21.091 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:17:21.092 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:17:21.092 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:17:21.092 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:17:21.092 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:17:21.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:17:21.526 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:17:22.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:17:22.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:17:22.528 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:17:22.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:17:22.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:17:22.529 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:17:23.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:17:24.012 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:17:24.012 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:17:24.012 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:17:24.013 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:17:24.013 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:17:24.013 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:17:24.559 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:17:25.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:17:25.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:17:25.561 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:17:25.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:17:25.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:17:25.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:17:26.014 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:17:27.016 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:17:27.016 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:17:27.017 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:17:27.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:17:27.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:17:27.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:17:27.643 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:17:28.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:17:28.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:17:28.645 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-25T23:17:28.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-25T23:17:28.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:17:28.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:17:29.044 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:17:30.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:17:30.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:17:30.046 INFO:tasks.workunit.client.0.smithi129.stdout:21 2024-03-25T23:17:30.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-25T23:17:30.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:17:30.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:17:30.367 INFO:tasks.workunit.client.0.smithi129.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2220983412,v1:127.0.0.1:6811/2220983412] [v2:127.0.0.1:6812/2220983412,v1:127.0.0.1:6813/2220983412] exists,up 4993326f-c2cb-4650-a57e-fbe3f3c536c4 2024-03-25T23:17:30.369 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:17:30.369 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:17:30.369 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:17:30.369 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-03-25T23:17:30.369 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:17:30.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:17:30.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:17:30.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-25T23:17:30.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:17:30.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-25T23:17:30.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:17:30.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:17:30.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:17:30.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:17:30.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-25T23:17:30.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-25T23:17:30.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:17:30.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:17:30.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:17:30.371 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:17:30.371 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:17:30.371 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:17:30.371 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:17:30.371 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:17:30.371 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:17:30.371 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:17:30.372 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:17:30.372 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:17:30.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:17:30.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:17:30.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:17:30.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:17:30.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:17:30.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:17:30.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:17:30.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:17:30.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:17:30.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-25T23:17:30.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-25T23:17:30.373 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:17:30.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=99b71be1-fdac-4168-ad45-da844f9a58ac 2024-03-25T23:17:30.374 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 99b71be1-fdac-4168-ad45-da844f9a58ac' 2024-03-25T23:17:30.374 INFO:tasks.workunit.client.0.smithi129.stdout:add osd2 99b71be1-fdac-4168-ad45-da844f9a58ac 2024-03-25T23:17:30.374 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:17:30.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAKBgJmkbDMFxAASI1QdzT+EuLUtX0jIOmY8Q== 2024-03-25T23:17:30.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAKBgJmkbDMFxAASI1QdzT+EuLUtX0jIOmY8Q=="}' 2024-03-25T23:17:30.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 99b71be1-fdac-4168-ad45-da844f9a58ac -i td/osd-scrub-repair/2/new.json 2024-03-25T23:17:30.797 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:17:30.816 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-25T23:17:30.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAKBgJmkbDMFxAASI1QdzT+EuLUtX0jIOmY8Q== --osd-uuid 99b71be1-fdac-4168-ad45-da844f9a58ac 2024-03-25T23:17:30.854 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:17:30.854+0000 7f2306c927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:17:30.899 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:17:30.898+0000 7f2306c927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:17:30.900 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:17:30.902+0000 7f2306c927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:17:30.901 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:17:30.902+0000 7f2306c927c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-25T23:17:30.901 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:17:30.902+0000 7f2306c927c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-25T23:17:30.901 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:17:30.902+0000 7f2306c927c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-25T23:17:30.989 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:17:30.990+0000 7f2306c927c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-25T23:17:37.819 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-25T23:17:37.819 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:17:37.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-25T23:17:37.820 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd2 key to auth repository 2024-03-25T23:17:37.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:17:38.216 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-25T23:17:38.216 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.2 2024-03-25T23:17:38.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:17:38.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:17:38.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:17:38.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:17:38.250 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:17:38.250+0000 7fdac99e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:17:38.288 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:17:38.290+0000 7fdac99e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:17:38.289 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:17:38.290+0000 7fdac99e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:17:38.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-25T23:17:38.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:17:38.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-25T23:17:38.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:17:38.567 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:17:38.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:17:38.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:17:38.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:17:38.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:17:38.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:17:38.846 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:17:38.846+0000 7fdac99e77c0 -1 Falling back to public interface 2024-03-25T23:17:38.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:17:39.871 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:17:39.871 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:17:39.871 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:17:39.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:17:39.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:17:39.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:17:40.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:17:41.191 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:17:41.191 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:17:41.191 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:17:41.192 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:17:41.192 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:17:41.192 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:17:41.529 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:17:41.558 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:17:41.558+0000 7fdac99e77c0 -1 osd.2 0 log_to_monitors true 2024-03-25T23:17:42.531 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:17:42.531 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:17:42.532 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:17:42.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:17:42.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:17:42.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:17:43.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:17:44.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:17:44.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:17:44.037 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:17:44.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:17:44.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:17:44.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:17:44.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:17:45.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:17:45.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:17:45.518 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:17:45.518 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:17:45.518 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:17:45.518 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:17:45.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:17:46.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:17:46.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:17:46.839 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:17:46.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:17:46.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:17:46.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:17:47.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:17:48.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:17:48.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:17:48.199 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:17:48.200 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:17:48.200 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:17:48.200 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:17:48.518 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:17:49.520 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:17:49.520 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:17:49.520 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:17:49.521 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:17:49.521 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:17:49.521 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:17:49.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:17:50.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:17:50.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:17:50.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:17:50.917 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:17:50.918 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:17:50.918 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:17:51.367 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:17:52.369 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:17:52.369 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:17:52.369 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:17:52.369 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:17:52.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:17:52.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:17:52.750 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:17:53.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:17:53.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:17:53.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:17:53.753 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:17:53.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:17:53.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:17:54.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:17:55.068 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:17:55.068 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:17:55.068 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:17:55.069 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:17:55.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:17:55.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:17:55.463 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:17:56.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:17:56.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:17:56.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:17:56.467 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:17:56.468 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:17:56.468 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:17:56.848 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:17:57.849 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:17:57.850 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:17:57.850 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:17:57.851 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:17:57.851 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:17:57.851 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:17:58.684 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:17:59.686 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:17:59.686 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:17:59.686 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:17:59.687 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:17:59.687 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:17:59.687 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:18:00.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:18:01.277 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:18:01.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:18:01.278 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:18:01.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:18:01.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:18:01.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:18:02.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:18:03.071 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:18:03.071 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:18:03.071 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:18:03.072 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:18:03.072 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:18:03.072 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:18:03.733 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:18:04.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:18:04.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:18:04.735 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:18:04.736 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:18:04.736 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:18:04.736 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:18:05.304 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:18:06.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:18:06.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:18:06.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:18:06.307 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:18:06.308 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:18:06.308 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:18:06.733 INFO:tasks.workunit.client.0.smithi129.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/714306193,v1:127.0.0.1:6819/714306193] [v2:127.0.0.1:6820/714306193,v1:127.0.0.1:6821/714306193] exists,up 99b71be1-fdac-4168-ad45-da844f9a58ac 2024-03-25T23:18:06.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:18:06.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:18:06.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:18:06.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:552: TEST_auto_repair_bluestore_failed: create_pool testpool 1 1 2024-03-25T23:18:06.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-03-25T23:18:07.157 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'testpool' created 2024-03-25T23:18:07.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T23:18:08.181 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:553: TEST_auto_repair_bluestore_failed: ceph osd pool set testpool size 2 2024-03-25T23:18:08.672 INFO:tasks.workunit.client.0.smithi129.stderr:set pool 1 size to 2 2024-03-25T23:18:08.694 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:554: TEST_auto_repair_bluestore_failed: wait_for_clean 2024-03-25T23:18:08.694 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:18:08.694 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:18:08.694 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:18:08.695 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:18:08.695 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:18:08.695 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:18:08.696 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:18:08.696 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:18:08.696 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:18:08.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:18:08.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:18:08.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:18:08.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:18:08.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:18:08.810 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:18:09.159 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:18:09.159 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:18:09.160 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-25T23:18:09.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:18:09.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:18:09.160 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:18:09.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836499 2024-03-25T23:18:09.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836499 2024-03-25T23:18:09.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499' 2024-03-25T23:18:09.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:18:09.326 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:18:09.482 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672971 2024-03-25T23:18:09.488 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672971 2024-03-25T23:18:09.488 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672971' 2024-03-25T23:18:09.488 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:18:09.488 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:18:09.762 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509444 2024-03-25T23:18:09.762 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509444 2024-03-25T23:18:09.762 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672971 2-64424509444' 2024-03-25T23:18:09.762 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:18:09.763 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836499 2024-03-25T23:18:09.763 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:18:09.764 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:18:09.764 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836499 2024-03-25T23:18:09.765 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:18:09.766 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836499 2024-03-25T23:18:09.766 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 21474836499 2024-03-25T23:18:09.766 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-03-25T23:18:09.766 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:18:10.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836499 2024-03-25T23:18:10.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:18:11.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:18:11.071 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:18:11.424 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836499 2024-03-25T23:18:11.424 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:18:11.424 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:18:11.424 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672971 2024-03-25T23:18:11.425 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:18:11.426 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672971 2024-03-25T23:18:11.426 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:18:11.427 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 42949672971 2024-03-25T23:18:11.428 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672971 2024-03-25T23:18:11.428 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2024-03-25T23:18:11.428 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:18:11.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672971 -lt 42949672971 2024-03-25T23:18:11.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:18:11.756 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:18:11.756 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509444 2024-03-25T23:18:11.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:18:11.758 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509444 2024-03-25T23:18:11.759 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:18:11.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509444 2024-03-25T23:18:11.760 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 64424509444 2024-03-25T23:18:11.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2024-03-25T23:18:11.761 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:18:12.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509444 -lt 64424509444 2024-03-25T23:18:12.091 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:18:12.091 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:18:12.092 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:18:12.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-25T23:18:12.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:18:12.435 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:18:12.436 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:18:12.436 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:18:12.436 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:18:12.436 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:18:12.436 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:18:12.764 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-25T23:18:12.764 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:18:12.765 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:18:12.765 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:18:13.122 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-25T23:18:13.122 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:18:13.122 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:18:13.122 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:557: TEST_auto_repair_bluestore_failed: local payload=ABCDEF 2024-03-25T23:18:13.122 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:558: TEST_auto_repair_bluestore_failed: echo ABCDEF 2024-03-25T23:18:13.122 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: seq 1 10 2024-03-25T23:18:13.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-25T23:18:13.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-03-25T23:18:13.271 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-25T23:18:13.271 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-03-25T23:18:13.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-25T23:18:13.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-03-25T23:18:13.547 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-25T23:18:13.547 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-03-25T23:18:13.682 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-25T23:18:13.682 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-03-25T23:18:13.824 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-25T23:18:13.824 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-03-25T23:18:13.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-25T23:18:13.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-03-25T23:18:14.134 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-25T23:18:14.134 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-03-25T23:18:14.292 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-25T23:18:14.293 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-03-25T23:18:14.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-25T23:18:14.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-03-25T23:18:14.623 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:566: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2024-03-25T23:18:14.623 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-25T23:18:14.624 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-25T23:18:14.624 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-25T23:18:14.624 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-25T23:18:14.625 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-25T23:18:14.625 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-25T23:18:14.625 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-25T23:18:14.953 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-25T23:18:14.953 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-25T23:18:14.953 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-25T23:18:15.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:566: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2024-03-25T23:18:15.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:18:15.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:18:15.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-25T23:18:15.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:18:15.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2024-03-25T23:18:15.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:18:15.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:18:15.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-25T23:18:15.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:18:15.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-25T23:18:15.298 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:18:15.298 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:18:15.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:18:15.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:18:15.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:18:15.606 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:18:15.606 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2024-03-25T23:18:15.606 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:18:15.606 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:18:15.606 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-25T23:18:15.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:18:15.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-25T23:18:15.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2024-03-25T23:18:17.086 INFO:tasks.workunit.client.0.smithi129.stdout:remove #1:ff7b1f36:::obj1:head# 2024-03-25T23:18:17.647 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:18:17.647 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:18:17.647 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:18:17.647 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-25T23:18:17.647 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:18:17.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-25T23:18:17.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:18:17.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:18:17.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:18:17.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:18:17.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-25T23:18:17.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-25T23:18:17.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:18:17.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:18:17.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:18:17.648 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:18:17.648 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:18:17.648 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:18:17.650 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:18:17.650 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:18:17.650 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:18:17.650 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:18:17.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:18:17.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:18:17.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:18:17.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:18:17.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:18:17.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:18:17.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:18:17.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:18:17.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:18:17.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-25T23:18:17.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-25T23:18:17.652 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-25T23:18:17.652 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.0 2024-03-25T23:18:17.652 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:18:17.652 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-25T23:18:17.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-25T23:18:17.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:18:17.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:18:17.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:18:17.688 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:18:17.690+0000 7f447b5517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:18:17.691 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:18:17.694+0000 7f447b5517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:18:17.693 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:18:17.694+0000 7f447b5517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:18:17.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-25T23:18:17.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:18:17.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T23:18:17.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:18:17.965 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:18:17.965 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:18:17.965 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:18:17.965 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-25T23:18:17.965 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:18:17.965 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:18:18.225 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:18:18.226+0000 7f447b5517c0 -1 Falling back to public interface 2024-03-25T23:18:18.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:18:19.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:18:19.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:18:19.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:18:19.347 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:18:19.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:18:19.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:18:19.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:18:20.652 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:18:20.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:18:20.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:18:20.653 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:18:20.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:18:20.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:18:20.995 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:18:21.033 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:18:21.038+0000 7f447b5517c0 -1 osd.0 21 log_to_monitors true 2024-03-25T23:18:21.997 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:18:21.997 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:18:21.997 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:18:21.997 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:18:21.997 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:18:21.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:18:22.405 INFO:tasks.workunit.client.0.smithi129.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/3205551734,v1:127.0.0.1:6803/3205551734] [v2:127.0.0.1:6804/3205551734,v1:127.0.0.1:6805/3205551734] exists,up f73ae165-97fa-4ffd-8fc9-39fa3de80b0b 2024-03-25T23:18:22.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:18:22.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:18:22.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:18:22.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:18:22.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:18:22.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:18:22.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:18:22.406 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:18:22.407 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:18:22.407 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:18:22.408 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:18:22.408 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:18:22.408 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:18:22.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:18:22.516 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:18:22.516 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:18:22.516 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:18:22.516 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:18:22.516 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:18:22.850 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:18:22.850 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:18:22.850 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-25T23:18:22.850 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:18:22.850 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:18:22.851 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:18:23.016 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-03-25T23:18:23.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-03-25T23:18:23.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-03-25T23:18:23.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:18:23.017 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:18:23.174 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672974 2024-03-25T23:18:23.174 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672974 2024-03-25T23:18:23.174 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672974' 2024-03-25T23:18:23.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:18:23.175 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:18:23.317 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509448 2024-03-25T23:18:23.318 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509448 2024-03-25T23:18:23.318 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672974 2-64424509448' 2024-03-25T23:18:23.318 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:18:23.318 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-03-25T23:18:23.318 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:18:23.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:18:23.320 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-03-25T23:18:23.320 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:18:23.321 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-03-25T23:18:23.321 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-03-25T23:18:23.321 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 107374182402 2024-03-25T23:18:23.321 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:18:23.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-03-25T23:18:23.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:18:24.633 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:18:24.634 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:18:24.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-03-25T23:18:24.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:18:24.988 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672974 2024-03-25T23:18:24.989 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:18:24.990 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:18:24.990 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672974 2024-03-25T23:18:24.990 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:18:24.992 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672974 2024-03-25T23:18:24.992 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-03-25T23:18:24.992 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 42949672974 2024-03-25T23:18:24.992 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:18:25.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672975 -lt 42949672974 2024-03-25T23:18:25.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:18:25.330 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509448 2024-03-25T23:18:25.331 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:18:25.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:18:25.333 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509448 2024-03-25T23:18:25.333 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:18:25.334 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509448 2024-03-25T23:18:25.334 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-03-25T23:18:25.334 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 64424509448 2024-03-25T23:18:25.335 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:18:25.644 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509448 -lt 64424509448 2024-03-25T23:18:25.645 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:18:25.645 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:18:25.645 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:18:26.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-25T23:18:26.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:18:26.040 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:18:26.041 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:18:26.041 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:18:26.041 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:18:26.041 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:18:26.041 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:18:26.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-25T23:18:26.344 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:18:26.344 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:18:26.344 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:18:26.720 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-25T23:18:26.720 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:18:26.720 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:18:26.720 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:568: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2024-03-25T23:18:26.721 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-25T23:18:26.721 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-25T23:18:26.721 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-25T23:18:26.722 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-25T23:18:26.722 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-25T23:18:26.722 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-25T23:18:26.722 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-25T23:18:27.068 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-25T23:18:27.069 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-25T23:18:27.069 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-25T23:18:27.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:568: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2024-03-25T23:18:27.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:18:27.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:18:27.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-25T23:18:27.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:18:27.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2024-03-25T23:18:27.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:18:27.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:18:27.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-25T23:18:27.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:18:27.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-25T23:18:27.410 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:18:27.410 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:18:27.410 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:18:27.410 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:18:27.410 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:18:27.718 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:18:27.718 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-03-25T23:18:27.718 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:18:27.718 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:18:27.718 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-25T23:18:27.718 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:18:27.718 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-25T23:18:27.718 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-03-25T23:18:29.237 INFO:tasks.workunit.client.0.smithi129.stdout:remove #1:104778fc:::obj2:head# 2024-03-25T23:18:29.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:18:29.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:18:29.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:18:29.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-25T23:18:29.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:18:29.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-25T23:18:29.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:18:29.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:18:29.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:18:29.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:18:29.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-25T23:18:29.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-25T23:18:29.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:18:29.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:18:29.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:18:29.777 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:18:29.777 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:18:29.777 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:18:29.777 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:18:29.777 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:18:29.777 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:18:29.777 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:18:29.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:18:29.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:18:29.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:18:29.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:18:29.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:18:29.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:18:29.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:18:29.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:18:29.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:18:29.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-25T23:18:29.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-25T23:18:29.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-25T23:18:29.780 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.0 2024-03-25T23:18:29.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:18:29.780 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-25T23:18:29.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-25T23:18:29.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:18:29.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:18:29.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:18:29.820 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:18:29.826+0000 7f46cf9147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:18:29.823 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:18:29.830+0000 7f46cf9147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:18:29.824 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:18:29.830+0000 7f46cf9147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:18:30.096 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-25T23:18:30.096 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:18:30.096 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T23:18:30.096 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:18:30.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:18:30.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:18:30.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:18:30.097 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-25T23:18:30.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:18:30.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:18:30.357 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:18:30.362+0000 7f46cf9147c0 -1 Falling back to public interface 2024-03-25T23:18:30.426 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:18:31.427 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:18:31.428 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:18:31.428 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:18:31.428 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:18:31.428 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:18:31.428 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:18:31.738 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:18:32.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:18:32.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:18:32.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:18:32.740 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:18:32.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:18:32.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:18:32.951 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:18:32.957+0000 7f46cf9147c0 -1 osd.0 26 log_to_monitors true 2024-03-25T23:18:33.050 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:18:34.052 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:18:34.052 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:18:34.052 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:18:34.052 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:18:34.053 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:18:34.053 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:18:34.394 INFO:tasks.workunit.client.0.smithi129.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/3657976166,v1:127.0.0.1:6803/3657976166] [v2:127.0.0.1:6804/3657976166,v1:127.0.0.1:6805/3657976166] exists,up f73ae165-97fa-4ffd-8fc9-39fa3de80b0b 2024-03-25T23:18:34.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:18:34.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:18:34.395 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:18:34.395 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:18:34.395 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:18:34.395 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:18:34.395 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:18:34.395 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:18:34.396 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:18:34.396 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:18:34.396 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:18:34.396 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:18:34.396 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:18:34.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:18:34.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:18:34.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:18:34.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:18:34.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:18:34.498 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:18:34.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:18:34.814 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:18:34.814 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-25T23:18:34.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:18:34.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:18:34.815 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:18:34.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-03-25T23:18:34.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-03-25T23:18:34.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882' 2024-03-25T23:18:34.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:18:34.964 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:18:35.109 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672978 2024-03-25T23:18:35.109 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672978 2024-03-25T23:18:35.109 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672978' 2024-03-25T23:18:35.110 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:18:35.110 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:18:35.253 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509451 2024-03-25T23:18:35.253 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509451 2024-03-25T23:18:35.253 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672978 2-64424509451' 2024-03-25T23:18:35.253 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:18:35.254 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018882 2024-03-25T23:18:35.254 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:18:35.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:18:35.255 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018882 2024-03-25T23:18:35.255 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:18:35.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-03-25T23:18:35.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-03-25T23:18:35.256 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 128849018882 2024-03-25T23:18:35.256 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:18:35.548 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 128849018882 2024-03-25T23:18:35.548 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:18:36.549 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:18:36.550 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:18:36.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-03-25T23:18:36.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:18:36.875 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672978 2024-03-25T23:18:36.875 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:18:36.876 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:18:36.877 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672978 2024-03-25T23:18:36.877 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:18:36.878 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672978 2024-03-25T23:18:36.879 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-03-25T23:18:36.879 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 42949672978 2024-03-25T23:18:36.879 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:18:37.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672978 -lt 42949672978 2024-03-25T23:18:37.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:18:37.217 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509451 2024-03-25T23:18:37.217 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:18:37.219 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:18:37.220 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509451 2024-03-25T23:18:37.220 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:18:37.221 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509451 2024-03-25T23:18:37.221 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-03-25T23:18:37.221 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 64424509451 2024-03-25T23:18:37.222 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:18:37.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509451 -lt 64424509451 2024-03-25T23:18:37.519 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:18:37.519 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:18:37.519 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:18:37.942 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-25T23:18:37.942 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:18:37.942 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:18:37.942 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:18:37.942 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:18:37.942 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:18:37.942 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:18:37.942 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:18:38.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-25T23:18:38.279 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:18:38.279 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:18:38.279 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:18:38.655 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-25T23:18:38.656 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:18:38.656 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:18:38.656 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2024-03-25T23:18:38.657 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-25T23:18:38.657 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-25T23:18:38.657 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-25T23:18:38.657 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-25T23:18:38.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2024-03-25T23:18:38.992 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:18:38.992 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:18:38.992 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-25T23:18:38.992 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:18:38.992 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2024-03-25T23:18:38.992 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:18:38.992 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:18:38.992 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-25T23:18:38.992 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:18:38.992 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-25T23:18:38.992 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:18:38.992 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:18:38.993 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:18:38.993 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:18:38.993 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:18:39.300 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:18:39.300 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2024-03-25T23:18:39.300 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:18:39.300 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:18:39.300 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-25T23:18:39.301 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:18:39.301 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:18:39.301 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2024-03-25T23:18:41.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:18:41.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:18:41.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:18:41.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-25T23:18:41.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:18:41.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:18:41.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:18:41.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:18:41.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:18:41.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:18:41.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-25T23:18:41.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-25T23:18:41.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:18:41.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:18:41.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:18:41.389 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:18:41.389 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:18:41.389 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:18:41.389 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:18:41.389 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:18:41.390 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:18:41.390 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:18:41.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:18:41.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:18:41.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:18:41.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:18:41.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:18:41.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:18:41.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:18:41.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:18:41.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:18:41.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-25T23:18:41.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-25T23:18:41.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-25T23:18:41.392 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.1 2024-03-25T23:18:41.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:18:41.393 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-25T23:18:41.393 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-25T23:18:41.393 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:18:41.393 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:18:41.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:18:41.427 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:18:41.429+0000 7f20be9c67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:18:41.429 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:18:41.433+0000 7f20be9c67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:18:41.431 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:18:41.433+0000 7f20be9c67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:18:41.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-25T23:18:41.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:18:41.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T23:18:41.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:18:41.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:18:41.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:18:41.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:18:41.713 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-25T23:18:41.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:18:41.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:18:41.969 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:18:41.973+0000 7f20be9c67c0 -1 Falling back to public interface 2024-03-25T23:18:42.055 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:18:43.056 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:18:43.056 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:18:43.056 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:18:43.057 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:18:43.057 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:18:43.057 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:18:43.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:18:44.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:18:44.421 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:18:44.421 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:18:44.421 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:18:44.421 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:18:44.421 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:18:44.594 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:18:44.597+0000 7f20be9c67c0 -1 osd.1 31 log_to_monitors true 2024-03-25T23:18:44.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:18:45.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:18:45.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:18:45.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:18:45.743 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:18:45.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:18:45.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:18:46.069 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1 up in weight 1 up_from 35 up_thru 35 down_at 32 last_clean_interval [10,31) [v2:127.0.0.1:6810/254084980,v1:127.0.0.1:6811/254084980] [v2:127.0.0.1:6812/254084980,v1:127.0.0.1:6813/254084980] exists,up 4993326f-c2cb-4650-a57e-fbe3f3c536c4 2024-03-25T23:18:46.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:18:46.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:18:46.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:18:46.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:18:46.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:18:46.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:18:46.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:18:46.069 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:18:46.070 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:18:46.070 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:18:46.070 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:18:46.070 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:18:46.070 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:18:46.170 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:18:46.171 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:18:46.171 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:18:46.171 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:18:46.171 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:18:46.171 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:18:46.473 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:18:46.473 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:18:46.473 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-25T23:18:46.474 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:18:46.474 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:18:46.474 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:18:46.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018885 2024-03-25T23:18:46.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018885 2024-03-25T23:18:46.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018885' 2024-03-25T23:18:46.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:18:46.660 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:18:46.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-03-25T23:18:46.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-03-25T23:18:46.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018885 1-150323855362' 2024-03-25T23:18:46.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:18:46.800 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:18:46.940 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509454 2024-03-25T23:18:46.940 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509454 2024-03-25T23:18:46.940 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018885 1-150323855362 2-64424509454' 2024-03-25T23:18:46.940 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:18:46.940 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018885 2024-03-25T23:18:46.940 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:18:46.941 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:18:46.941 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018885 2024-03-25T23:18:46.942 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:18:46.943 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018885 2024-03-25T23:18:46.943 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018885' 2024-03-25T23:18:46.943 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 128849018885 2024-03-25T23:18:46.943 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:18:47.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018884 -lt 128849018885 2024-03-25T23:18:47.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:18:48.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:18:48.238 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:18:48.538 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018884 -lt 128849018885 2024-03-25T23:18:48.538 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:18:49.540 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-25T23:18:49.540 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:18:49.849 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018886 -lt 128849018885 2024-03-25T23:18:49.849 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:18:49.850 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855362 2024-03-25T23:18:49.850 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:18:49.852 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:18:49.852 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855362 2024-03-25T23:18:49.853 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:18:49.854 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-03-25T23:18:49.854 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855362' 2024-03-25T23:18:49.854 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 150323855362 2024-03-25T23:18:49.854 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:18:50.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-03-25T23:18:50.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:18:50.202 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509454 2024-03-25T23:18:50.202 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:18:50.204 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:18:50.205 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509454 2024-03-25T23:18:50.205 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:18:50.206 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509454 2024-03-25T23:18:50.206 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-03-25T23:18:50.206 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 64424509454 2024-03-25T23:18:50.206 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:18:50.547 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509455 -lt 64424509454 2024-03-25T23:18:50.547 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:18:50.548 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:18:50.548 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:18:50.993 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-25T23:18:50.993 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:18:50.994 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:18:50.994 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:18:50.994 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:18:50.994 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:18:50.994 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:18:50.994 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:18:51.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-25T23:18:51.338 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:18:51.338 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:18:51.338 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:18:51.690 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-25T23:18:51.690 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:18:51.690 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:18:51.690 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:571: TEST_auto_repair_bluestore_failed: get_pg testpool obj1 2024-03-25T23:18:51.690 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-03-25T23:18:51.690 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-03-25T23:18:51.691 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-03-25T23:18:51.691 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-25T23:18:52.003 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:571: TEST_auto_repair_bluestore_failed: local pgid=1.0 2024-03-25T23:18:52.004 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:572: TEST_auto_repair_bluestore_failed: get_primary testpool obj1 2024-03-25T23:18:52.004 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-25T23:18:52.004 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-03-25T23:18:52.005 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-03-25T23:18:52.005 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-25T23:18:52.335 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:572: TEST_auto_repair_bluestore_failed: local primary=1 2024-03-25T23:18:52.336 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:573: TEST_auto_repair_bluestore_failed: get_last_scrub_stamp 1.0 2024-03-25T23:18:52.336 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-25T23:18:52.336 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:18:52.336 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:18:52.336 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-25T23:18:52.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:573: TEST_auto_repair_bluestore_failed: local last_scrub_stamp=2024-03-25T23:18:07.094103+0000 2024-03-25T23:18:52.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:574: TEST_auto_repair_bluestore_failed: ceph tell 1.0 schedule-deep-scrub 2024-03-25T23:18:52.754 INFO:tasks.workunit.client.0.smithi129.stdout:{ 2024-03-25T23:18:52.754 INFO:tasks.workunit.client.0.smithi129.stdout: "deep": true, 2024-03-25T23:18:52.754 INFO:tasks.workunit.client.0.smithi129.stdout: "must": false, 2024-03-25T23:18:52.755 INFO:tasks.workunit.client.0.smithi129.stdout: "stamp": "2024-03-11T23:17:12.761182+0000" 2024-03-25T23:18:52.755 INFO:tasks.workunit.client.0.smithi129.stdout:} 2024-03-25T23:18:52.768 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:577: TEST_auto_repair_bluestore_failed: wait_for_scrub 1.0 2024-03-25T23:18:07.094103+0000 2024-03-25T23:18:52.769 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-03-25T23:18:52.769 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-25T23:18:07.094103+0000 2024-03-25T23:18:52.769 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-25T23:18:52.769 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-25T23:18:52.769 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:18:52.769 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-25T23:18:52.770 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-25T23:18:52.770 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:18:52.770 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:18:52.770 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-25T23:18:53.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:18:07.094103+0000 '>' 2024-03-25T23:18:07.094103+0000 2024-03-25T23:18:53.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:18:54.068 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:18:54.068 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:18:54.069 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-25T23:18:54.069 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-25T23:18:54.069 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:18:54.070 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:18:54.070 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-25T23:18:54.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:18:07.094103+0000 '>' 2024-03-25T23:18:07.094103+0000 2024-03-25T23:18:54.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:18:55.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:18:55.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:18:55.390 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-25T23:18:55.391 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-25T23:18:55.391 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:18:55.391 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:18:55.391 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-25T23:18:55.705 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:18:07.094103+0000 '>' 2024-03-25T23:18:07.094103+0000 2024-03-25T23:18:55.705 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:18:56.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:18:56.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:18:56.707 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-25T23:18:56.707 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-25T23:18:56.707 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:18:56.707 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:18:56.707 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-25T23:18:57.026 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:18:53.965871+0000 '>' 2024-03-25T23:18:07.094103+0000 2024-03-25T23:18:57.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-25T23:18:57.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:578: TEST_auto_repair_bluestore_failed: wait_for_clean 2024-03-25T23:18:57.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:18:57.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:18:57.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:18:57.028 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:18:57.028 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:18:57.028 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:18:57.028 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:18:57.028 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:18:57.029 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:18:57.122 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:18:57.122 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:18:57.122 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:18:57.122 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:18:57.122 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:18:57.122 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:18:57.440 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:18:57.440 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:18:57.440 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-25T23:18:57.440 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:18:57.440 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:18:57.440 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:18:57.600 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018888 2024-03-25T23:18:57.601 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018888 2024-03-25T23:18:57.601 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018888' 2024-03-25T23:18:57.601 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:18:57.601 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:18:57.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855365 2024-03-25T23:18:57.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855365 2024-03-25T23:18:57.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018888 1-150323855365' 2024-03-25T23:18:57.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:18:57.734 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:18:57.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509458 2024-03-25T23:18:57.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509458 2024-03-25T23:18:57.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018888 1-150323855365 2-64424509458' 2024-03-25T23:18:57.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:18:57.872 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018888 2024-03-25T23:18:57.872 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:18:57.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:18:57.873 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018888 2024-03-25T23:18:57.874 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:18:57.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018888 2024-03-25T23:18:57.875 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018888' 2024-03-25T23:18:57.875 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 128849018888 2024-03-25T23:18:57.875 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:18:58.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018887 -lt 128849018888 2024-03-25T23:18:58.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:18:59.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:18:59.179 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:18:59.480 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018889 -lt 128849018888 2024-03-25T23:18:59.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:18:59.481 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855365 2024-03-25T23:18:59.482 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:18:59.483 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:18:59.484 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855365 2024-03-25T23:18:59.484 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:18:59.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855365 2024-03-25T23:18:59.486 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855365' 2024-03-25T23:18:59.486 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 150323855365 2024-03-25T23:18:59.486 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:18:59.799 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855365 -lt 150323855365 2024-03-25T23:18:59.799 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:18:59.800 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509458 2024-03-25T23:18:59.800 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:18:59.801 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:18:59.802 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509458 2024-03-25T23:18:59.802 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:18:59.804 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509458 2024-03-25T23:18:59.804 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-03-25T23:18:59.804 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 64424509458 2024-03-25T23:18:59.804 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:19:00.159 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509458 -lt 64424509458 2024-03-25T23:19:00.159 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:19:00.160 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:19:00.160 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:19:00.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-25T23:19:00.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:19:00.542 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:19:00.542 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:19:00.542 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:19:00.542 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:19:00.542 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:19:00.542 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:19:00.909 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-25T23:19:00.910 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:19:00.910 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:19:00.910 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:19:01.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-25T23:19:01.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:19:01.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:19:01.266 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:579: TEST_auto_repair_bluestore_failed: flush_pg_stats 2024-03-25T23:19:01.266 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:19:01.266 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:19:01.617 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:19:01.618 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:19:01.618 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-25T23:19:01.618 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:19:01.618 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:19:01.618 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:19:01.798 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018890 2024-03-25T23:19:01.799 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018890 2024-03-25T23:19:01.799 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018890' 2024-03-25T23:19:01.799 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:19:01.799 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:19:01.940 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855367 2024-03-25T23:19:01.940 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855367 2024-03-25T23:19:01.940 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018890 1-150323855367' 2024-03-25T23:19:01.940 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:19:01.941 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:19:02.075 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509459 2024-03-25T23:19:02.075 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509459 2024-03-25T23:19:02.075 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018890 1-150323855367 2-64424509459' 2024-03-25T23:19:02.075 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:19:02.075 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018890 2024-03-25T23:19:02.075 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:19:02.076 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:19:02.076 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018890 2024-03-25T23:19:02.076 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:19:02.077 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018890 2024-03-25T23:19:02.077 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018890' 2024-03-25T23:19:02.078 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 128849018890 2024-03-25T23:19:02.078 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:19:02.375 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018889 -lt 128849018890 2024-03-25T23:19:02.375 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:19:03.376 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:19:03.377 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:19:03.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018890 -lt 128849018890 2024-03-25T23:19:03.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:19:03.690 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855367 2024-03-25T23:19:03.690 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:19:03.692 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:19:03.692 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855367 2024-03-25T23:19:03.693 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:19:03.694 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855367 2024-03-25T23:19:03.694 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855367' 2024-03-25T23:19:03.694 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 150323855367 2024-03-25T23:19:03.695 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:19:04.031 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855367 -lt 150323855367 2024-03-25T23:19:04.031 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:19:04.031 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509459 2024-03-25T23:19:04.032 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:19:04.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:19:04.033 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509459 2024-03-25T23:19:04.033 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:19:04.035 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509459 2024-03-25T23:19:04.035 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-03-25T23:19:04.036 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 64424509459 2024-03-25T23:19:04.036 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:19:04.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509460 -lt 64424509459 2024-03-25T23:19:04.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:580: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-03-25T23:19:04.373 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:18:53.729+0000 7f20a2950640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-03-25T23:19:04.373 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:18:53.729+0000 7f20a2950640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-03-25T23:19:04.373 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:18:53.729+0000 7f20a2950640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-03-25T23:19:04.374 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:18:53.729+0000 7f20a2950640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-03-25T23:19:04.374 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:18:53.729+0000 7f20a2950640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-25T23:19:04.374 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:18:53.961+0000 7f20a2950640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber: scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2024-03-25T23:19:04.374 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:18:53.961+0000 7f20a2950640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-03-25T23:19:04.374 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:18:53.961+0000 7f20a2950640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-25T23:19:04.374 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:18:53.961+0000 7f20a2950640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: REQ_SCRUB ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-03-25T23:19:04.374 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:581: TEST_auto_repair_bluestore_failed: grep -q 'scrub_finish.*still present after re-scrub' td/osd-scrub-repair/osd.1.log 2024-03-25T23:19:04.375 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:582: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-03-25T23:19:04.658 INFO:tasks.workunit.client.0.smithi129.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-25T23:19:04.658 INFO:tasks.workunit.client.0.smithi129.stdout:1.0 10 0 0 0 0 63 0 0 10 0 10 active+clean+inconsistent+failed_repair 2024-03-25T23:18:53.966122+0000 21'10 36:102 [1,0] 1 [1,0] 1 21'10 2024-03-25T23:18:53.965871+0000 21'10 2024-03-25T23:18:53.965871+0000 0 1 periodic scrub scheduled @ 2024-03-26T23:18:53.965871+0000 10 0 2024-03-25T23:19:04.658 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:19:04.658 INFO:tasks.workunit.client.0.smithi129.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-25T23:19:04.659 INFO:tasks.workunit.client.0.smithi129.stderr:dumped pgs 2024-03-25T23:19:04.674 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:583: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-03-25T23:19:04.674 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:583: TEST_auto_repair_bluestore_failed: grep -q '^1.0.*+failed_repair' 2024-03-25T23:19:04.942 INFO:tasks.workunit.client.0.smithi129.stderr:dumped pgs 2024-03-25T23:19:04.958 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:587: TEST_auto_repair_bluestore_failed: get_not_primary testpool obj1 2024-03-25T23:19:04.958 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-25T23:19:04.958 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-03-25T23:19:04.959 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool obj1 2024-03-25T23:19:04.959 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-25T23:19:04.959 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-03-25T23:19:04.959 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-03-25T23:19:04.959 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-25T23:19:05.247 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-25T23:19:05.248 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool obj1 2024-03-25T23:19:05.248 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-25T23:19:05.554 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:587: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 list-attrs 2024-03-25T23:19:05.554 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:19:05.554 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:19:05.554 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-25T23:19:05.554 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:19:05.555 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 list-attrs 2024-03-25T23:19:05.555 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:19:05.555 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:19:05.555 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-25T23:19:05.555 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:19:05.555 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-25T23:19:05.555 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:19:05.555 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:19:05.556 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:19:05.556 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:19:05.556 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:19:05.865 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:19:05.865 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 list-attrs 2024-03-25T23:19:05.865 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:19:05.865 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:19:05.866 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-25T23:19:05.866 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:19:05.866 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-25T23:19:05.866 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 list-attrs 2024-03-25T23:19:06.138 INFO:tasks.workunit.client.0.smithi129.stdout:_ 2024-03-25T23:19:06.138 INFO:tasks.workunit.client.0.smithi129.stdout:snapset 2024-03-25T23:19:06.430 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:19:06.430 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:19:06.430 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:19:06.430 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-25T23:19:06.430 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:19:06.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-25T23:19:06.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:19:06.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:19:06.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:19:06.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:19:06.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-25T23:19:06.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-25T23:19:06.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:19:06.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:19:06.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:19:06.431 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:19:06.431 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:19:06.431 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:19:06.432 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:19:06.432 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:19:06.432 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:19:06.432 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:19:06.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:19:06.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:19:06.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:19:06.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:19:06.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:19:06.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:19:06.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:19:06.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:19:06.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:19:06.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-25T23:19:06.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-25T23:19:06.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-25T23:19:06.434 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.0 2024-03-25T23:19:06.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:19:06.435 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-25T23:19:06.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-25T23:19:06.436 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:19:06.437 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:19:06.437 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:19:06.469 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:19:06.473+0000 7fd164fe27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:19:06.472 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:19:06.477+0000 7fd164fe27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:19:06.473 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:19:06.477+0000 7fd164fe27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:19:06.754 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-25T23:19:06.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:19:06.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T23:19:06.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:19:06.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:19:06.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:19:06.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:19:06.755 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-25T23:19:06.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:19:06.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:19:07.033 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:19:07.037+0000 7fd164fe27c0 -1 Falling back to public interface 2024-03-25T23:19:07.049 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:19:08.051 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:19:08.051 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:19:08.051 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:19:08.051 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:19:08.051 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:19:08.051 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:19:08.367 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:19:09.368 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:19:09.369 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:19:09.369 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:19:09.369 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:19:09.369 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:19:09.369 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:19:09.686 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:19:10.187 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:19:10.189+0000 7fd164fe27c0 -1 osd.0 36 log_to_monitors true 2024-03-25T23:19:10.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:19:10.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:19:10.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:19:10.690 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:19:10.690 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:19:10.690 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:19:11.091 INFO:tasks.workunit.client.0.smithi129.stderr:osd.0 up in weight 1 up_from 40 up_thru 32 down_at 37 last_clean_interval [30,36) [v2:127.0.0.1:6802/3673318484,v1:127.0.0.1:6803/3673318484] [v2:127.0.0.1:6804/3673318484,v1:127.0.0.1:6805/3673318484] exists,up f73ae165-97fa-4ffd-8fc9-39fa3de80b0b 2024-03-25T23:19:11.091 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:19:11.091 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:19:11.091 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:19:11.091 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:19:11.091 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:19:11.091 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:19:11.091 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:19:11.091 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:19:11.092 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:19:11.092 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:19:11.092 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:19:11.092 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:19:11.092 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:19:11.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:19:11.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:19:11.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:19:11.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:19:11.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:19:11.199 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:19:11.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:19:11.689 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:19:11.690 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-25T23:19:11.690 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:19:11.690 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:19:11.690 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:19:11.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-03-25T23:19:11.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-03-25T23:19:11.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691842' 2024-03-25T23:19:11.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:19:11.862 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:19:11.999 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855370 2024-03-25T23:19:11.999 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855370 2024-03-25T23:19:11.999 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691842 1-150323855370' 2024-03-25T23:19:11.999 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:19:11.999 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:19:12.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509462 2024-03-25T23:19:12.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509462 2024-03-25T23:19:12.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691842 1-150323855370 2-64424509462' 2024-03-25T23:19:12.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:19:12.143 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691842 2024-03-25T23:19:12.143 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:19:12.143 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:19:12.144 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691842 2024-03-25T23:19:12.144 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:19:12.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-03-25T23:19:12.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691842' 2024-03-25T23:19:12.145 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 171798691842 2024-03-25T23:19:12.146 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:19:12.442 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 171798691842 2024-03-25T23:19:12.442 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:19:13.444 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:19:13.445 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:19:13.764 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691842 -lt 171798691842 2024-03-25T23:19:13.765 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:19:13.765 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:19:13.765 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855370 2024-03-25T23:19:13.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:19:13.768 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855370 2024-03-25T23:19:13.768 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:19:13.769 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855370 2024-03-25T23:19:13.769 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855370' 2024-03-25T23:19:13.769 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 150323855370 2024-03-25T23:19:13.770 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:19:14.109 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855370 -lt 150323855370 2024-03-25T23:19:14.109 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:19:14.110 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:19:14.110 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509462 2024-03-25T23:19:14.111 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:19:14.112 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509462 2024-03-25T23:19:14.112 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:19:14.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509462 2024-03-25T23:19:14.114 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-03-25T23:19:14.114 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 64424509462 2024-03-25T23:19:14.114 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:19:14.456 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509463 -lt 64424509462 2024-03-25T23:19:14.457 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:19:14.457 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:19:14.457 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:19:14.867 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-25T23:19:14.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:19:14.869 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:19:14.869 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:19:14.869 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:19:14.869 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:19:14.869 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:19:14.869 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:19:15.191 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-25T23:19:15.192 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:19:15.192 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:19:15.192 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:19:15.545 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-25T23:19:15.545 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:19:15.545 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:19:15.545 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:588: TEST_auto_repair_bluestore_failed: rados --pool testpool get obj1 td/osd-scrub-repair/COPY 2024-03-25T23:19:15.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:589: TEST_auto_repair_bluestore_failed: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-25T23:19:15.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:590: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-03-25T23:19:15.582 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:18:53.729+0000 7f20a2950640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-03-25T23:19:15.582 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:18:53.729+0000 7f20a2950640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-03-25T23:19:15.582 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:18:53.729+0000 7f20a2950640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-03-25T23:19:15.583 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:18:53.729+0000 7f20a2950640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-03-25T23:19:15.583 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:18:53.729+0000 7f20a2950640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-25T23:19:15.583 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:18:53.961+0000 7f20a2950640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber: scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2024-03-25T23:19:15.583 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:18:53.961+0000 7f20a2950640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-03-25T23:19:15.583 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:18:53.961+0000 7f20a2950640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-25T23:19:15.583 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:18:53.961+0000 7f20a2950640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: REQ_SCRUB ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-03-25T23:19:15.584 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:593: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2024-03-25T23:19:15.584 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-25T23:19:15.584 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-25T23:19:15.584 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-25T23:19:15.584 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-25T23:19:15.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:593: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 remove 2024-03-25T23:19:15.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:19:15.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:19:15.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-25T23:19:15.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:19:15.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 remove 2024-03-25T23:19:15.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:19:15.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:19:15.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-25T23:19:15.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:19:15.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-25T23:19:15.873 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:19:15.874 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:19:15.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:19:15.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:19:15.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:19:16.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:19:16.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 remove 2024-03-25T23:19:16.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:19:16.183 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:19:16.183 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-25T23:19:16.183 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:19:16.183 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:19:16.183 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 remove 2024-03-25T23:19:17.665 INFO:tasks.workunit.client.0.smithi129.stderr:Error getting attr on : 1.0_head,#1:104778fc:::obj2:head#, (61) No data available 2024-03-25T23:19:17.666 INFO:tasks.workunit.client.0.smithi129.stdout:remove #1:104778fc:::obj2:head# 2024-03-25T23:19:18.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:19:18.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:19:18.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:19:18.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-25T23:19:18.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:19:18.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:19:18.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:19:18.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:19:18.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:19:18.208 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:19:18.208 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-25T23:19:18.208 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-25T23:19:18.208 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:19:18.208 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:19:18.208 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:19:18.208 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:19:18.208 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:19:18.208 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:19:18.208 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:19:18.208 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:19:18.208 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:19:18.208 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:19:18.210 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:19:18.210 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:19:18.210 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:19:18.210 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:19:18.210 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:19:18.210 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:19:18.210 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:19:18.210 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:19:18.210 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:19:18.210 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-25T23:19:18.210 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-25T23:19:18.210 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-25T23:19:18.211 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.1 2024-03-25T23:19:18.211 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:19:18.211 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-25T23:19:18.212 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-25T23:19:18.213 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:19:18.213 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:19:18.213 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:19:18.254 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:19:18.257+0000 7fc688bca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:19:18.257 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:19:18.261+0000 7fc688bca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:19:18.261 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:19:18.261+0000 7fc688bca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:19:18.523 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-25T23:19:18.523 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:19:18.523 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T23:19:18.523 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:19:18.523 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:19:18.523 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:19:18.523 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:19:18.523 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-25T23:19:18.523 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:19:18.523 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:19:18.797 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:19:18.801+0000 7fc688bca7c0 -1 Falling back to public interface 2024-03-25T23:19:18.894 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:19:19.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:19:19.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:19:19.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:19:19.896 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:19:19.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:19:19.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:19:20.216 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:19:21.218 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:19:21.226 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:19:21.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:19:21.227 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:19:21.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:19:21.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:19:21.446 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:19:21.449+0000 7fc688bca7c0 -1 osd.1 41 log_to_monitors true 2024-03-25T23:19:21.555 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:19:22.557 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:19:22.557 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:19:22.557 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:19:22.557 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:19:22.557 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:19:22.557 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:19:22.950 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1 up in weight 1 up_from 45 up_thru 45 down_at 42 last_clean_interval [35,41) [v2:127.0.0.1:6810/3500622161,v1:127.0.0.1:6811/3500622161] [v2:127.0.0.1:6812/3500622161,v1:127.0.0.1:6813/3500622161] exists,up 4993326f-c2cb-4650-a57e-fbe3f3c536c4 2024-03-25T23:19:22.950 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:19:22.950 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:19:22.950 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:19:22.951 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:19:22.951 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:19:22.951 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:19:22.951 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:19:22.951 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:19:22.952 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:19:22.952 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:19:22.952 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:19:22.952 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:19:22.952 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:19:23.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:19:23.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:19:23.063 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:19:23.063 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:19:23.063 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:19:23.063 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:19:23.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:19:23.389 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:19:23.390 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-25T23:19:23.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:19:23.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:19:23.390 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:19:23.574 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691845 2024-03-25T23:19:23.574 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691845 2024-03-25T23:19:23.574 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691845' 2024-03-25T23:19:23.574 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:19:23.574 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:19:23.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528322 2024-03-25T23:19:23.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528322 2024-03-25T23:19:23.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691845 1-193273528322' 2024-03-25T23:19:23.717 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:19:23.717 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:19:23.859 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509466 2024-03-25T23:19:23.859 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509466 2024-03-25T23:19:23.859 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691845 1-193273528322 2-64424509466' 2024-03-25T23:19:23.859 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:19:23.860 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691845 2024-03-25T23:19:23.860 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:19:23.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:19:23.861 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691845 2024-03-25T23:19:23.861 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:19:23.862 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691845 2024-03-25T23:19:23.862 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691845' 2024-03-25T23:19:23.862 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 171798691845 2024-03-25T23:19:23.863 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:19:24.170 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691844 -lt 171798691845 2024-03-25T23:19:24.170 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:19:25.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:19:25.173 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:19:25.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691845 -lt 171798691845 2024-03-25T23:19:25.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:19:25.498 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:19:25.498 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-193273528322 2024-03-25T23:19:25.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:19:25.501 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:19:25.501 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-193273528322 2024-03-25T23:19:25.502 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528322 2024-03-25T23:19:25.502 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 193273528322' 2024-03-25T23:19:25.502 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 193273528322 2024-03-25T23:19:25.504 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:19:25.828 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528322 -lt 193273528322 2024-03-25T23:19:25.828 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:19:25.829 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509466 2024-03-25T23:19:25.829 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:19:25.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:19:25.831 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509466 2024-03-25T23:19:25.831 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:19:25.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509466 2024-03-25T23:19:25.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509466' 2024-03-25T23:19:25.833 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 64424509466 2024-03-25T23:19:25.833 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:19:26.143 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509466 -lt 64424509466 2024-03-25T23:19:26.144 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:19:26.144 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:19:26.145 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:19:26.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-25T23:19:26.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:19:26.546 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:19:26.546 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:19:26.546 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:19:26.546 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:19:26.547 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:19:26.547 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:19:26.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-25T23:19:26.916 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:19:26.916 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:19:26.916 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:19:27.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-25T23:19:27.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:19:27.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:19:27.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:594: TEST_auto_repair_bluestore_failed: repair 1.0 2024-03-25T23:19:27.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-03-25T23:19:27.262 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-03-25T23:19:27.262 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-25T23:19:27.262 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:19:27.262 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:19:27.262 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-25T23:19:27.557 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-25T23:18:53.965871+0000 2024-03-25T23:19:27.558 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-03-25T23:19:27.835 INFO:tasks.workunit.client.0.smithi129.stderr:instructing pg 1.0 on osd.1 to repair 2024-03-25T23:19:27.851 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-03-25T23:18:53.965871+0000 2024-03-25T23:19:27.851 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-03-25T23:19:27.851 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-25T23:18:53.965871+0000 2024-03-25T23:19:27.851 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-25T23:19:27.851 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-25T23:19:27.851 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:19:27.852 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-25T23:19:27.852 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-25T23:19:27.852 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:19:27.852 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-25T23:19:27.852 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:19:28.161 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:18:53.965871+0000 '>' 2024-03-25T23:18:53.965871+0000 2024-03-25T23:19:28.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:19:29.164 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:19:29.164 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:19:29.164 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-25T23:19:29.164 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-25T23:19:29.165 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:19:29.165 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:19:29.165 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-25T23:19:29.487 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:18:53.965871+0000 '>' 2024-03-25T23:18:53.965871+0000 2024-03-25T23:19:29.487 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:19:30.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:19:30.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:19:30.489 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-25T23:19:30.489 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-25T23:19:30.490 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:19:30.490 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:19:30.490 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-25T23:19:30.846 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:18:53.965871+0000 '>' 2024-03-25T23:18:53.965871+0000 2024-03-25T23:19:30.846 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:19:31.847 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:19:31.847 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:19:31.848 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-25T23:19:31.848 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-25T23:19:31.848 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:19:31.849 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:19:31.849 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-25T23:19:32.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:18:53.965871+0000 '>' 2024-03-25T23:18:53.965871+0000 2024-03-25T23:19:32.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:19:33.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:19:33.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:19:33.170 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-25T23:19:33.170 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-25T23:19:33.170 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:19:33.171 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:19:33.171 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-25T23:19:33.491 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:19:28.529987+0000 '>' 2024-03-25T23:18:53.965871+0000 2024-03-25T23:19:33.491 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-25T23:19:33.492 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:595: TEST_auto_repair_bluestore_failed: sleep 2 2024-03-25T23:19:35.492 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:597: TEST_auto_repair_bluestore_failed: flush_pg_stats 2024-03-25T23:19:35.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:19:35.493 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:19:35.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:19:35.805 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:19:35.805 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-25T23:19:35.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:19:35.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:19:35.806 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:19:36.004 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691849 2024-03-25T23:19:36.004 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691849 2024-03-25T23:19:36.004 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691849' 2024-03-25T23:19:36.004 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:19:36.005 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:19:36.147 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528325 2024-03-25T23:19:36.147 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528325 2024-03-25T23:19:36.148 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691849 1-193273528325' 2024-03-25T23:19:36.148 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:19:36.148 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:19:36.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509469 2024-03-25T23:19:36.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509469 2024-03-25T23:19:36.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691849 1-193273528325 2-64424509469' 2024-03-25T23:19:36.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:19:36.279 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691849 2024-03-25T23:19:36.279 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:19:36.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:19:36.280 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691849 2024-03-25T23:19:36.280 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:19:36.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691849 2024-03-25T23:19:36.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691849' 2024-03-25T23:19:36.282 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 171798691849 2024-03-25T23:19:36.282 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:19:36.571 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691847 -lt 171798691849 2024-03-25T23:19:36.571 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:19:37.573 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:19:37.574 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:19:37.895 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691849 -lt 171798691849 2024-03-25T23:19:37.895 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:19:37.896 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-193273528325 2024-03-25T23:19:37.896 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:19:37.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:19:37.899 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-193273528325 2024-03-25T23:19:37.899 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:19:37.901 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528325 2024-03-25T23:19:37.901 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 193273528325 2024-03-25T23:19:37.901 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 193273528325' 2024-03-25T23:19:37.902 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:19:38.213 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528325 -lt 193273528325 2024-03-25T23:19:38.213 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:19:38.214 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509469 2024-03-25T23:19:38.214 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:19:38.216 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:19:38.217 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509469 2024-03-25T23:19:38.217 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:19:38.218 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 64424509469 2024-03-25T23:19:38.219 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509469 2024-03-25T23:19:38.219 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509469' 2024-03-25T23:19:38.219 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:19:38.566 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509469 -lt 64424509469 2024-03-25T23:19:38.566 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:598: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-03-25T23:19:38.915 INFO:tasks.workunit.client.0.smithi129.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-25T23:19:38.915 INFO:tasks.workunit.client.0.smithi129.stdout:1.0 9 0 0 0 0 63 0 0 10 0 10 active+clean 2024-03-25T23:19:28.530249+0000 21'10 46:139 [1,0] 1 [1,0] 1 21'10 2024-03-25T23:19:28.529987+0000 21'10 2024-03-25T23:19:28.529987+0000 0 1 periodic scrub scheduled @ 2024-03-26T23:19:28.529987+0000 9 0 2024-03-25T23:19:38.916 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:19:38.916 INFO:tasks.workunit.client.0.smithi129.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-25T23:19:38.916 INFO:tasks.workunit.client.0.smithi129.stderr:dumped pgs 2024-03-25T23:19:38.933 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:599: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-03-25T23:19:38.933 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:599: TEST_auto_repair_bluestore_failed: grep -q -e '^1.0.* active+clean ' -e '^1.0.* active+clean+wait ' 2024-03-25T23:19:39.227 INFO:tasks.workunit.client.0.smithi129.stderr:dumped pgs 2024-03-25T23:19:39.244 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:600: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-03-25T23:19:39.247 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:18:53.729+0000 7f20a2950640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-03-25T23:19:39.247 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:18:53.729+0000 7f20a2950640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-03-25T23:19:39.247 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:18:53.729+0000 7f20a2950640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-03-25T23:19:39.248 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:18:53.729+0000 7f20a2950640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-03-25T23:19:39.248 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:18:53.729+0000 7f20a2950640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-25T23:19:39.248 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:18:53.961+0000 7f20a2950640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber: scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2024-03-25T23:19:39.248 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:18:53.961+0000 7f20a2950640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-03-25T23:19:39.248 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:18:53.961+0000 7f20a2950640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-25T23:19:39.248 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:18:53.961+0000 7f20a2950640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: REQ_SCRUB ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-03-25T23:19:39.248 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:19:28.525+0000 7fc66cb54640 10 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=10 ec=16/16 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] MUST_REPAIR planned AUTO_REPAIR MUST_DEEP_SCRUB MUST_SCRUB planned REQ_SCRUB] scrubber: scrub_finish before flags: AUTO_REPAIR REQ_SCRUB. repair state: repair. deep_scrub_on_error: 0 2024-03-25T23:19:39.248 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:19:28.525+0000 7fc66cb54640 10 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=10 ec=16/16 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-03-25T23:19:39.248 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:19:28.525+0000 7fc66cb54640 15 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=16/16 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-03-25T23:19:39.248 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:19:28.525+0000 7fc66cb54640 20 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=16/16 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish All may be fixed 2024-03-25T23:19:39.248 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:19:28.525+0000 7fc66cb54640 19 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=16/16 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-25T23:19:39.249 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-25T23:19:39.249 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-25T23:19:39.249 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T23:19:39.249 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-25T23:19:39.250 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:19:39.250 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:19:39.250 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:19:39.250 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:19:39.250 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:19:39.559 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:19:39.560 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T23:19:39.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T23:19:39.561 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T23:19:39.563 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T23:19:39.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T23:19:39.564 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T23:19:39.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:19:39.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T23:19:39.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T23:19:39.566 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:19:39.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T23:19:39.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T23:19:39.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-25T23:19:39.592 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T23:19:39.592 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:19:39.592 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:19:39.592 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-25T23:19:39.595 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T23:19:39.595 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T23:19:39.595 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-25T23:19:39.595 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-25T23:19:39.595 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-25T23:19:39.595 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-25T23:19:39.595 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-25T23:19:39.595 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T23:19:39.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-25T23:19:39.596 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:19:39.596 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:19:39.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:19:39.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:19:39.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:19:39.597 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:19:39.598 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T23:19:39.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T23:19:39.599 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T23:19:39.600 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T23:19:39.601 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T23:19:39.601 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T23:19:39.602 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:19:39.602 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T23:19:39.603 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T23:19:39.603 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:19:39.604 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T23:19:39.605 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T23:19:39.605 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-25T23:19:39.606 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T23:19:39.607 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:19:39.607 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:19:39.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-25T23:19:39.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T23:19:39.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T23:19:39.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-25T23:19:39.610 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-25T23:19:39.610 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:19:39.610 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:19:39.610 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61636 2024-03-25T23:19:39.612 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-25T23:19:39.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-25T23:19:39.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-25T23:19:39.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-25T23:19:39.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_failed_norecov td/osd-scrub-repair 2024-03-25T23:19:39.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:604: TEST_auto_repair_bluestore_failed_norecov: local dir=td/osd-scrub-repair 2024-03-25T23:19:39.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:605: TEST_auto_repair_bluestore_failed_norecov: local poolname=testpool 2024-03-25T23:19:39.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:608: TEST_auto_repair_bluestore_failed_norecov: run_mon td/osd-scrub-repair a 2024-03-25T23:19:39.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-25T23:19:39.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-25T23:19:39.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-25T23:19:39.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-25T23:19:39.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-25T23:19:39.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-25T23:19:40.128 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-25T23:19:40.128 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:19:40.128 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:19:40.129 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:19:40.129 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:19:40.129 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:19:40.129 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:19:40.129 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-25T23:19:40.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-25T23:19:40.350 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-25T23:19:40.350 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T23:19:40.350 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T23:19:40.350 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-25T23:19:40.351 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-25T23:19:40.351 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T23:19:40.351 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T23:19:40.351 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T23:19:40.352 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T23:19:40.352 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:19:40.352 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:19:40.352 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-25T23:19:40.352 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T23:19:40.353 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get fsid 2024-03-25T23:19:40.472 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-25T23:19:40.473 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T23:19:40.473 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T23:19:40.473 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-25T23:19:40.473 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-25T23:19:40.474 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T23:19:40.474 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T23:19:40.474 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T23:19:40.474 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T23:19:40.474 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:19:40.475 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:19:40.475 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-25T23:19:40.475 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T23:19:40.476 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get mon_host 2024-03-25T23:19:40.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:609: TEST_auto_repair_bluestore_failed_norecov: run_mgr td/osd-scrub-repair x 2024-03-25T23:19:40.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-25T23:19:40.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-25T23:19:40.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-25T23:19:40.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-25T23:19:40.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-25T23:19:40.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-25T23:19:41.465 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-25T23:19:41.465 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:19:41.465 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:19:41.466 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:19:41.466 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:19:41.466 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:19:41.466 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:19:41.467 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T23:19:41.468 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T23:19:41.502 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:612: TEST_auto_repair_bluestore_failed_norecov: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-25T23:19:41.503 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: seq 0 2 2024-03-25T23:19:41.506 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-03-25T23:19:41.506 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:19:41.506 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:19:41.506 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:19:41.506 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-25T23:19:41.506 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:19:41.506 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-25T23:19:41.506 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:19:41.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:19:41.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:19:41.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:19:41.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-25T23:19:41.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-25T23:19:41.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:19:41.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:19:41.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:19:41.507 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:19:41.507 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:19:41.507 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:19:41.507 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:19:41.507 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:19:41.508 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:19:41.508 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:19:41.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:19:41.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:19:41.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:19:41.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:19:41.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:19:41.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:19:41.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:19:41.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:19:41.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:19:41.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:19:41.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:19:41.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-25T23:19:41.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-25T23:19:41.509 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:19:41.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=216868d9-b284-4562-9481-c9eb9f7570c1 2024-03-25T23:19:41.509 INFO:tasks.workunit.client.0.smithi129.stdout:add osd0 216868d9-b284-4562-9481-c9eb9f7570c1 2024-03-25T23:19:41.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 216868d9-b284-4562-9481-c9eb9f7570c1' 2024-03-25T23:19:41.510 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:19:41.526 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCNBgJmpU2/HxAAgHirtcGSGqzlWwimPQ9vyQ== 2024-03-25T23:19:41.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCNBgJmpU2/HxAAgHirtcGSGqzlWwimPQ9vyQ=="}' 2024-03-25T23:19:41.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 216868d9-b284-4562-9481-c9eb9f7570c1 -i td/osd-scrub-repair/0/new.json 2024-03-25T23:19:41.792 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:19:41.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-25T23:19:41.811 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCNBgJmpU2/HxAAgHirtcGSGqzlWwimPQ9vyQ== --osd-uuid 216868d9-b284-4562-9481-c9eb9f7570c1 2024-03-25T23:19:41.842 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:19:41.845+0000 7f2d5ae287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:19:41.859 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:19:41.865+0000 7f2d5ae287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:19:41.860 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:19:41.865+0000 7f2d5ae287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:19:41.861 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:19:41.865+0000 7f2d5ae287c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-25T23:19:41.861 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:19:41.865+0000 7f2d5ae287c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-25T23:19:41.861 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:19:41.865+0000 7f2d5ae287c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-25T23:19:41.951 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:19:41.957+0000 7f2d5ae287c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-25T23:19:48.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-25T23:19:48.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:19:48.299 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd0 key to auth repository 2024-03-25T23:19:48.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-25T23:19:48.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:19:48.683 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.0 2024-03-25T23:19:48.683 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-25T23:19:48.684 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:19:48.684 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:19:48.684 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:19:48.685 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:19:48.715 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:19:48.720+0000 7f36054877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:19:48.717 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:19:48.720+0000 7f36054877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:19:48.719 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:19:48.724+0000 7f36054877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:19:48.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-25T23:19:48.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:19:48.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T23:19:48.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:19:48.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:19:48.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:19:48.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:19:48.974 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:19:48.974 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:19:48.974 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:19:49.273 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:19:49.276+0000 7f36054877c0 -1 Falling back to public interface 2024-03-25T23:19:49.306 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:19:50.308 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:19:50.308 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:19:50.309 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:19:50.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:19:50.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:19:50.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:19:50.629 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:19:51.596 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:19:51.600+0000 7f36054877c0 -1 osd.0 0 log_to_monitors true 2024-03-25T23:19:51.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:19:51.632 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:19:51.632 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:19:51.632 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:19:51.632 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:19:51.633 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:19:52.030 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:19:53.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:19:53.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:19:53.033 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:19:53.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:19:53.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:19:53.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:19:53.374 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:19:54.376 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:19:54.376 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:19:54.377 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:19:54.377 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:19:54.377 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:19:54.377 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:19:54.682 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:19:55.684 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:19:55.684 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:19:55.684 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:19:55.685 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:19:55.685 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:19:55.685 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:19:55.992 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:19:56.994 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:19:56.994 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:19:56.994 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:19:56.995 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:19:56.995 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:19:56.995 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:19:57.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:19:58.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:19:58.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:19:58.344 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:19:58.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:19:58.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:19:58.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:19:58.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:19:59.666 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:19:59.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:19:59.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:19:59.667 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:19:59.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:19:59.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:19:59.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:20:00.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:20:00.989 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:20:00.989 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:20:00.989 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:20:00.990 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:20:00.990 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:20:01.335 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:20:02.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:20:02.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:20:02.337 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:20:02.338 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:20:02.338 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:20:02.338 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:20:02.662 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:20:03.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:20:03.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:20:03.665 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:20:03.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:20:03.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:20:03.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:20:03.982 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:20:04.984 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:20:04.984 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:20:04.984 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:20:04.985 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:20:04.985 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:20:04.985 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:20:05.684 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:20:06.686 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:20:06.686 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:20:06.686 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:20:06.686 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:20:06.687 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:20:06.687 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:20:07.187 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:20:08.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:20:08.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:20:08.189 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:20:08.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:20:08.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:20:08.190 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:20:08.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:20:09.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:20:09.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:20:09.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:20:09.689 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:20:09.690 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:20:09.690 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:20:10.249 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:20:11.250 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:20:11.251 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:20:11.251 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:20:11.251 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:20:11.252 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:20:11.252 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:20:11.980 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:20:12.982 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:20:12.982 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:20:12.983 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:20:12.983 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:20:12.983 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:20:12.983 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:20:13.514 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:20:14.516 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:20:14.516 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:20:14.517 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:20:14.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:20:14.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:20:14.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:20:15.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:20:16.354 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:20:16.354 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:20:16.354 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:20:16.355 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:20:16.355 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:20:16.355 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:20:16.706 INFO:tasks.workunit.client.0.smithi129.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/975481672,v1:127.0.0.1:6803/975481672] [v2:127.0.0.1:6804/975481672,v1:127.0.0.1:6805/975481672] exists,up 216868d9-b284-4562-9481-c9eb9f7570c1 2024-03-25T23:20:16.709 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:20:16.709 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:20:16.709 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:20:16.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-03-25T23:20:16.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:20:16.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:20:16.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:20:16.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-25T23:20:16.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:20:16.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:20:16.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:20:16.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:20:16.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:20:16.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:20:16.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-25T23:20:16.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-25T23:20:16.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:20:16.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:20:16.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:20:16.711 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:20:16.711 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:20:16.711 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:20:16.711 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:20:16.711 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:20:16.711 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:20:16.711 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:20:16.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:20:16.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:20:16.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:20:16.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:20:16.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:20:16.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:20:16.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:20:16.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:20:16.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:20:16.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:20:16.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:20:16.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-25T23:20:16.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-25T23:20:16.713 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:20:16.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3511f3d0-5053-4ee7-a03a-3c7ebf69bf89 2024-03-25T23:20:16.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 3511f3d0-5053-4ee7-a03a-3c7ebf69bf89' 2024-03-25T23:20:16.713 INFO:tasks.workunit.client.0.smithi129.stdout:add osd1 3511f3d0-5053-4ee7-a03a-3c7ebf69bf89 2024-03-25T23:20:16.713 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:20:16.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCwBgJmUd8iLBAAUaW+7u5EsF0i7BY89afQTg== 2024-03-25T23:20:16.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCwBgJmUd8iLBAAUaW+7u5EsF0i7BY89afQTg=="}' 2024-03-25T23:20:16.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3511f3d0-5053-4ee7-a03a-3c7ebf69bf89 -i td/osd-scrub-repair/1/new.json 2024-03-25T23:20:17.147 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:20:17.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-25T23:20:17.167 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCwBgJmUd8iLBAAUaW+7u5EsF0i7BY89afQTg== --osd-uuid 3511f3d0-5053-4ee7-a03a-3c7ebf69bf89 2024-03-25T23:20:17.200 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:20:17.204+0000 7faeca1437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:20:17.240 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:20:17.244+0000 7faeca1437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:20:17.241 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:20:17.244+0000 7faeca1437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:20:17.242 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:20:17.244+0000 7faeca1437c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-25T23:20:17.242 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:20:17.244+0000 7faeca1437c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-25T23:20:17.242 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:20:17.244+0000 7faeca1437c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-25T23:20:17.322 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:20:17.328+0000 7faeca1437c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-25T23:20:24.083 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-25T23:20:24.083 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:20:24.085 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd1 key to auth repository 2024-03-25T23:20:24.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-25T23:20:24.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:20:24.466 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.1 2024-03-25T23:20:24.466 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-25T23:20:24.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:20:24.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:20:24.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:20:24.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:20:24.500 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:20:24.504+0000 7f0488c837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:20:24.528 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:20:24.532+0000 7f0488c837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:20:24.529 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:20:24.532+0000 7f0488c837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:20:24.792 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-25T23:20:24.793 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:20:24.793 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T23:20:24.793 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:20:24.793 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:20:24.793 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:20:24.793 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:20:24.794 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:20:24.794 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:20:24.794 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:20:25.081 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:20:25.084+0000 7f0488c837c0 -1 Falling back to public interface 2024-03-25T23:20:25.147 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:20:26.148 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:20:26.148 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:20:26.149 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:20:26.149 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:20:26.149 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:20:26.149 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:20:26.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:20:27.466 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:20:27.466 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:20:27.466 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:20:27.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:20:27.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:20:27.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:20:27.725 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:20:27.728+0000 7f0488c837c0 -1 osd.1 0 log_to_monitors true 2024-03-25T23:20:27.786 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:20:28.787 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:20:28.788 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:20:28.788 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:20:28.788 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:20:28.788 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:20:28.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:20:29.146 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:20:30.147 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:20:30.148 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:20:30.148 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:20:30.148 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:20:30.149 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:20:30.149 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:20:30.589 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:20:31.592 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:20:31.592 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:20:31.592 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:20:31.593 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:20:31.593 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:20:31.593 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:20:31.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:20:32.912 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:20:32.912 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:20:32.912 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:20:32.913 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:20:32.913 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:20:32.913 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:20:33.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:20:34.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:20:34.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:20:34.230 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:20:34.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:20:34.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:20:34.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:20:34.594 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:20:35.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:20:35.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:20:35.596 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:20:35.597 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:20:35.597 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:20:35.597 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:20:35.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:20:36.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:20:36.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:20:36.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:20:36.910 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:20:36.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:20:36.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:20:37.277 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:20:38.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:20:38.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:20:38.280 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:20:38.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:20:38.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:20:38.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:20:38.669 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:20:39.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:20:39.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:20:39.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:20:39.671 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:20:39.672 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:20:39.672 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:20:39.995 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:20:40.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:20:40.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:20:40.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:20:40.999 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:20:40.999 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:20:41.000 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:20:41.324 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:20:42.325 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:20:42.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:20:42.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:20:42.326 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:20:42.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:20:42.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:20:43.184 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:20:44.185 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:20:44.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:20:44.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:20:44.186 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:20:44.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:20:44.187 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:20:45.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:20:46.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:20:46.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:20:46.267 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:20:46.268 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:20:46.268 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:20:46.268 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:20:47.073 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:20:48.075 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:20:48.075 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:20:48.075 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:20:48.076 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:20:48.076 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:20:48.076 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:20:48.445 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:20:49.446 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:20:49.447 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:20:49.447 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:20:49.447 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:20:49.447 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:20:49.447 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:20:50.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:20:51.258 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:20:51.258 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:20:51.258 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:20:51.259 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:20:51.259 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:20:51.259 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:20:51.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:20:52.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:20:52.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:20:52.836 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:20:52.837 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:20:52.837 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:20:52.837 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:20:53.218 INFO:tasks.workunit.client.0.smithi129.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3227463865,v1:127.0.0.1:6811/3227463865] [v2:127.0.0.1:6812/3227463865,v1:127.0.0.1:6813/3227463865] exists,up 3511f3d0-5053-4ee7-a03a-3c7ebf69bf89 2024-03-25T23:20:53.221 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:20:53.221 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:20:53.221 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:20:53.221 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-03-25T23:20:53.221 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:20:53.221 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:20:53.221 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:20:53.221 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-25T23:20:53.221 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:20:53.222 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-25T23:20:53.222 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:20:53.222 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:20:53.222 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:20:53.222 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:20:53.222 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-25T23:20:53.222 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-25T23:20:53.222 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:20:53.222 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:20:53.222 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:20:53.222 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:20:53.222 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:20:53.222 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:20:53.223 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:20:53.223 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:20:53.223 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:20:53.224 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:20:53.224 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:20:53.224 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:20:53.224 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:20:53.224 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:20:53.224 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:20:53.224 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:20:53.224 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:20:53.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:20:53.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:20:53.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:20:53.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:20:53.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-25T23:20:53.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-25T23:20:53.225 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:20:53.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4423d416-c50d-4730-ae71-534464c463f7 2024-03-25T23:20:53.226 INFO:tasks.workunit.client.0.smithi129.stdout:add osd2 4423d416-c50d-4730-ae71-534464c463f7 2024-03-25T23:20:53.226 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 4423d416-c50d-4730-ae71-534464c463f7' 2024-03-25T23:20:53.226 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:20:53.244 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDVBgJmyMfhDhAAmx46u5anYIqwFIGY09Pu5g== 2024-03-25T23:20:53.244 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDVBgJmyMfhDhAAmx46u5anYIqwFIGY09Pu5g=="}' 2024-03-25T23:20:53.244 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4423d416-c50d-4730-ae71-534464c463f7 -i td/osd-scrub-repair/2/new.json 2024-03-25T23:20:53.636 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:20:53.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-25T23:20:53.655 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDVBgJmyMfhDhAAmx46u5anYIqwFIGY09Pu5g== --osd-uuid 4423d416-c50d-4730-ae71-534464c463f7 2024-03-25T23:20:53.686 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:20:53.687+0000 7f985a6777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:20:53.730 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:20:53.731+0000 7f985a6777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:20:53.731 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:20:53.735+0000 7f985a6777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:20:53.732 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:20:53.735+0000 7f985a6777c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-25T23:20:53.732 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:20:53.735+0000 7f985a6777c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-25T23:20:53.732 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:20:53.735+0000 7f985a6777c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-25T23:20:53.801 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:20:53.803+0000 7f985a6777c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-25T23:21:00.516 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-25T23:21:00.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:21:00.518 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-25T23:21:00.518 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd2 key to auth repository 2024-03-25T23:21:00.519 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:21:00.941 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.2 2024-03-25T23:21:00.941 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-25T23:21:00.942 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:21:00.942 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:21:00.942 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:21:00.943 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:21:00.975 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:21:00.978+0000 7f9f49bcd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:21:01.019 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:21:01.022+0000 7f9f49bcd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:21:01.021 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:21:01.022+0000 7f9f49bcd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:21:01.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-25T23:21:01.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:21:01.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-25T23:21:01.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:21:01.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:21:01.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:21:01.240 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:21:01.241 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:21:01.241 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:21:01.241 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:21:01.566 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:21:01.604 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:21:01.606+0000 7f9f49bcd7c0 -1 Falling back to public interface 2024-03-25T23:21:02.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:21:02.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:21:02.568 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:21:02.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:21:02.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:21:02.569 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:21:02.871 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:21:03.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:21:03.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:21:03.874 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:21:03.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:21:03.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:21:03.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:21:04.191 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:21:04.241 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:21:04.242+0000 7f9f49bcd7c0 -1 osd.2 0 log_to_monitors true 2024-03-25T23:21:05.193 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:21:05.193 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:21:05.193 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:21:05.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:21:05.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:21:05.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:21:05.512 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:21:06.514 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:21:06.514 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:21:06.515 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:21:06.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:21:06.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:21:06.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:21:06.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:21:07.928 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:21:07.928 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:21:07.929 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:21:07.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:21:07.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:21:07.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:21:08.422 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:21:09.424 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:21:09.424 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:21:09.425 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:21:09.425 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:21:09.425 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:21:09.425 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:21:09.744 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:21:10.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:21:10.747 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:21:10.747 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:21:10.747 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:21:10.747 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:21:10.747 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:21:11.059 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:21:12.061 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:21:12.061 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:21:12.061 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:21:12.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:21:12.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:21:12.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:21:12.442 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:21:13.444 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:21:13.444 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:21:13.444 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:21:13.445 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:21:13.445 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:21:13.445 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:21:13.763 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:21:14.765 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:21:14.765 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:21:14.765 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:21:14.766 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:21:14.766 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:21:14.766 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:21:15.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:21:16.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:21:16.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:21:16.175 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:21:16.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:21:16.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:21:16.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:21:16.486 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:21:17.487 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:21:17.488 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:21:17.488 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:21:17.488 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:21:17.488 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:21:17.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:21:17.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:21:18.804 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:21:18.804 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:21:18.805 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:21:18.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:21:18.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:21:18.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:21:19.804 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:21:20.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:21:20.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:21:20.806 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:21:20.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:21:20.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:21:20.807 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:21:21.616 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:21:22.617 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:21:22.618 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:21:22.618 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:21:22.618 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:21:22.619 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:21:22.619 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:21:23.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:21:24.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:21:24.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:21:24.707 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:21:24.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:21:24.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:21:24.708 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:21:25.593 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:21:26.595 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:21:26.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:21:26.596 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:21:26.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:21:26.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:21:26.597 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:21:27.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:21:28.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:21:28.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:21:28.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:21:28.389 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:21:28.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:21:28.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:21:29.212 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:21:30.215 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:21:30.215 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:21:30.215 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:21:30.216 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:21:30.216 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:21:30.216 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:21:30.548 INFO:tasks.workunit.client.0.smithi129.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1484465923,v1:127.0.0.1:6819/1484465923] [v2:127.0.0.1:6820/1484465923,v1:127.0.0.1:6821/1484465923] exists,up 4423d416-c50d-4730-ae71-534464c463f7 2024-03-25T23:21:30.549 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:21:30.549 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:21:30.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:21:30.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:617: TEST_auto_repair_bluestore_failed_norecov: create_pool testpool 1 1 2024-03-25T23:21:30.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-03-25T23:21:30.983 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'testpool' created 2024-03-25T23:21:31.001 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T23:21:32.004 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed_norecov: ceph osd pool set testpool size 2 2024-03-25T23:21:32.638 INFO:tasks.workunit.client.0.smithi129.stderr:set pool 1 size to 2 2024-03-25T23:21:32.658 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:619: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2024-03-25T23:21:32.658 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:21:32.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:21:32.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:21:32.659 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:21:32.660 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:21:32.660 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:21:32.660 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:21:32.660 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:21:32.660 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:21:32.769 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:21:32.769 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:21:32.769 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:21:32.769 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:21:32.769 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:21:32.769 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:21:33.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:21:33.172 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:21:33.172 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-25T23:21:33.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:21:33.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:21:33.173 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:21:33.321 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836503 2024-03-25T23:21:33.321 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836503 2024-03-25T23:21:33.321 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503' 2024-03-25T23:21:33.321 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:21:33.321 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:21:33.465 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672972 2024-03-25T23:21:33.465 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672972 2024-03-25T23:21:33.465 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672972' 2024-03-25T23:21:33.465 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:21:33.465 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:21:33.669 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509446 2024-03-25T23:21:33.669 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509446 2024-03-25T23:21:33.669 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672972 2-64424509446' 2024-03-25T23:21:33.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:21:33.670 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836503 2024-03-25T23:21:33.670 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:21:33.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:21:33.672 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836503 2024-03-25T23:21:33.672 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:21:33.673 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836503 2024-03-25T23:21:33.674 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 21474836503 2024-03-25T23:21:33.674 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-03-25T23:21:33.674 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:21:34.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836503 2024-03-25T23:21:34.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:21:35.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:21:35.033 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:21:35.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836503 2024-03-25T23:21:35.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:21:35.348 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:21:35.348 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672972 2024-03-25T23:21:35.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:21:35.350 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672972 2024-03-25T23:21:35.350 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:21:35.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672972 2024-03-25T23:21:35.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-03-25T23:21:35.352 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 42949672972 2024-03-25T23:21:35.352 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:21:35.685 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672972 -lt 42949672972 2024-03-25T23:21:35.685 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:21:35.687 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509446 2024-03-25T23:21:35.687 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:21:35.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:21:35.689 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509446 2024-03-25T23:21:35.689 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:21:35.690 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509446 2024-03-25T23:21:35.691 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 64424509446 2024-03-25T23:21:35.691 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-03-25T23:21:35.691 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:21:36.018 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509446 -lt 64424509446 2024-03-25T23:21:36.019 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:21:36.019 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:21:36.019 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:21:36.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-25T23:21:36.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:21:36.416 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:21:36.416 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:21:36.416 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:21:36.416 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:21:36.416 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:21:36.416 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:21:36.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-25T23:21:36.727 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:21:36.728 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:21:36.728 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:21:37.072 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-03-25T23:21:37.072 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-03-25T23:21:37.072 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-25T23:21:37.072 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-03-25T23:21:37.072 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:21:37.072 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-25T23:21:37.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:21:37.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:21:37.174 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:21:37.174 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:21:37.174 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:21:37.174 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:21:37.175 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:21:37.175 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:21:37.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-25T23:21:37.486 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:21:37.486 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:21:37.487 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:21:37.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-03-25T23:21:37.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-03-25T23:21:37.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-25T23:21:37.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-25T23:21:37.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-25T23:21:37.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-25T23:21:37.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-25T23:21:37.864 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-25T23:21:37.864 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-25T23:21:38.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-25T23:21:38.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-25T23:21:38.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-25T23:21:38.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:21:38.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-25T23:21:38.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:21:38.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:21:38.462 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:21:38.462 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:21:38.462 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:21:38.462 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:21:38.462 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:21:38.463 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:21:38.787 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-25T23:21:38.787 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:21:38.788 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:21:38.788 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:21:39.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-03-25T23:21:39.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-03-25T23:21:39.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-25T23:21:39.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-25T23:21:39.143 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-25T23:21:39.143 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-25T23:21:39.143 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-25T23:21:39.143 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-25T23:21:39.143 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-25T23:21:39.540 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-25T23:21:39.540 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-25T23:21:39.540 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-25T23:21:39.540 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:21:39.540 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-25T23:21:39.941 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:21:39.941 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:21:39.941 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:21:39.942 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:21:39.942 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:21:39.942 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:21:39.942 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:21:39.942 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:21:40.314 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-25T23:21:40.315 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:21:40.315 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:21:40.315 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:21:40.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-25T23:21:40.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:21:40.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:21:40.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:622: TEST_auto_repair_bluestore_failed_norecov: local payload=ABCDEF 2024-03-25T23:21:40.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:623: TEST_auto_repair_bluestore_failed_norecov: echo ABCDEF 2024-03-25T23:21:40.704 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: seq 1 10 2024-03-25T23:21:40.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-25T23:21:40.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-03-25T23:21:40.862 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-25T23:21:40.862 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-03-25T23:21:41.057 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-25T23:21:41.057 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-03-25T23:21:41.357 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-25T23:21:41.357 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-03-25T23:21:41.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-25T23:21:41.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-03-25T23:21:41.658 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-25T23:21:41.658 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-03-25T23:21:41.808 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-25T23:21:41.808 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-03-25T23:21:41.959 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-25T23:21:41.959 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-03-25T23:21:42.118 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-25T23:21:42.118 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-03-25T23:21:42.269 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-25T23:21:42.269 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-03-25T23:21:42.419 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:632: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2024-03-25T23:21:42.420 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-25T23:21:42.420 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-25T23:21:42.421 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-25T23:21:42.421 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-25T23:21:42.421 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-25T23:21:42.421 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-25T23:21:42.421 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-25T23:21:42.746 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-25T23:21:42.746 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-25T23:21:42.746 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-25T23:21:43.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:632: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2024-03-25T23:21:43.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:21:43.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:21:43.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-25T23:21:43.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:21:43.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2024-03-25T23:21:43.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:21:43.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:21:43.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-25T23:21:43.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:21:43.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-25T23:21:43.070 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:21:43.070 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:21:43.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:21:43.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:21:43.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:21:43.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:21:43.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2024-03-25T23:21:43.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:21:43.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:21:43.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-25T23:21:43.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:21:43.380 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-25T23:21:43.380 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2024-03-25T23:21:44.967 INFO:tasks.workunit.client.0.smithi129.stdout:remove #1:ff7b1f36:::obj1:head# 2024-03-25T23:21:45.503 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:21:45.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:21:45.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:21:45.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-25T23:21:45.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:21:45.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-25T23:21:45.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:21:45.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:21:45.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:21:45.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:21:45.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-25T23:21:45.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-25T23:21:45.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:21:45.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:21:45.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:21:45.505 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:21:45.505 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:21:45.505 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:21:45.507 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:21:45.507 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:21:45.507 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:21:45.507 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:21:45.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:21:45.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:21:45.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:21:45.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:21:45.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:21:45.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:21:45.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:21:45.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:21:45.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:21:45.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-25T23:21:45.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-25T23:21:45.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-25T23:21:45.508 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.0 2024-03-25T23:21:45.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:21:45.509 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-25T23:21:45.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-25T23:21:45.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:21:45.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:21:45.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:21:45.543 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:21:45.545+0000 7f55de39c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:21:45.546 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:21:45.549+0000 7f55de39c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:21:45.548 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:21:45.553+0000 7f55de39c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:21:45.818 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-25T23:21:45.818 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:21:45.818 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T23:21:45.818 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:21:45.819 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:21:45.819 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:21:45.819 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:21:45.819 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-25T23:21:45.819 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:21:45.819 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:21:46.082 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:21:46.085+0000 7f55de39c7c0 -1 Falling back to public interface 2024-03-25T23:21:46.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:21:47.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:21:47.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:21:47.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:21:47.139 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:21:47.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:21:47.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:21:47.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:21:48.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:21:48.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:21:48.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:21:48.461 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:21:48.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:21:48.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:21:48.837 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:21:48.867 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:21:48.869+0000 7f55de39c7c0 -1 osd.0 21 log_to_monitors true 2024-03-25T23:21:49.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:21:49.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:21:49.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:21:49.839 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:21:49.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:21:49.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:21:50.191 INFO:tasks.workunit.client.0.smithi129.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/1411651927,v1:127.0.0.1:6803/1411651927] [v2:127.0.0.1:6804/1411651927,v1:127.0.0.1:6805/1411651927] exists,up 216868d9-b284-4562-9481-c9eb9f7570c1 2024-03-25T23:21:50.191 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:21:50.191 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:21:50.192 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:21:50.192 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:21:50.192 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:21:50.192 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:21:50.192 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:21:50.193 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:21:50.193 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:21:50.193 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:21:50.193 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:21:50.194 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:21:50.194 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:21:50.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:21:50.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:21:50.304 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:21:50.304 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:21:50.304 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:21:50.304 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:21:50.749 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:21:50.749 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:21:50.749 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-25T23:21:50.749 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:21:50.749 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:21:50.750 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:21:50.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-03-25T23:21:50.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-03-25T23:21:50.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-03-25T23:21:50.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:21:50.920 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:21:51.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672976 2024-03-25T23:21:51.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672976 2024-03-25T23:21:51.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672976' 2024-03-25T23:21:51.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:21:51.062 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:21:51.204 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509451 2024-03-25T23:21:51.204 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509451 2024-03-25T23:21:51.204 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672976 2-64424509451' 2024-03-25T23:21:51.204 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:21:51.205 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-03-25T23:21:51.205 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:21:51.205 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:21:51.206 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-03-25T23:21:51.206 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:21:51.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-03-25T23:21:51.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-03-25T23:21:51.207 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 107374182402 2024-03-25T23:21:51.208 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:21:51.491 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-03-25T23:21:51.491 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:21:52.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:21:52.494 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:21:52.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-03-25T23:21:52.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:21:52.798 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672976 2024-03-25T23:21:52.798 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:21:52.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:21:52.801 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672976 2024-03-25T23:21:52.801 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:21:52.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672976 2024-03-25T23:21:52.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-03-25T23:21:52.802 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 42949672976 2024-03-25T23:21:52.802 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:21:53.116 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672976 2024-03-25T23:21:53.116 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:21:53.117 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:21:53.117 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509451 2024-03-25T23:21:53.118 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:21:53.119 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509451 2024-03-25T23:21:53.119 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:21:53.120 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509451 2024-03-25T23:21:53.120 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-03-25T23:21:53.120 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 64424509451 2024-03-25T23:21:53.120 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:21:53.428 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509451 -lt 64424509451 2024-03-25T23:21:53.428 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:21:53.429 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:21:53.429 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:21:53.812 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-25T23:21:53.813 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:21:53.813 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:21:53.814 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:21:53.814 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:21:53.814 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:21:53.814 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:21:53.814 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:21:54.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-25T23:21:54.140 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:21:54.140 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:21:54.140 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:21:54.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-25T23:21:54.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:21:54.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:21:54.494 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:633: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2024-03-25T23:21:54.494 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-25T23:21:54.494 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-25T23:21:54.495 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-25T23:21:54.495 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-25T23:21:54.845 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:633: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj1 rm-attr _ 2024-03-25T23:21:54.846 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:21:54.846 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:21:54.846 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-25T23:21:54.846 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:21:54.846 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj1 rm-attr _ 2024-03-25T23:21:54.846 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:21:54.846 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:21:54.846 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-25T23:21:54.846 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:21:54.846 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-25T23:21:54.846 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:21:54.847 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:21:54.847 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:21:54.847 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:21:54.847 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:21:55.154 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:21:55.155 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 rm-attr _ 2024-03-25T23:21:55.155 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:21:55.155 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:21:55.155 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-25T23:21:55.155 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:21:55.155 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:21:55.155 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 rm-attr _ 2024-03-25T23:21:57.293 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:21:57.294 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:21:57.294 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:21:57.294 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-25T23:21:57.294 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:21:57.294 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:21:57.294 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:21:57.294 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:21:57.294 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:21:57.294 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:21:57.294 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-25T23:21:57.295 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-25T23:21:57.295 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:21:57.295 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:21:57.295 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:21:57.295 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:21:57.295 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:21:57.295 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:21:57.297 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:21:57.297 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:21:57.297 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:21:57.297 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:21:57.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:21:57.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:21:57.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:21:57.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:21:57.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:21:57.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:21:57.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:21:57.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:21:57.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:21:57.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-25T23:21:57.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-25T23:21:57.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-25T23:21:57.298 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.1 2024-03-25T23:21:57.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:21:57.299 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-25T23:21:57.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-25T23:21:57.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:21:57.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:21:57.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:21:57.333 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:21:57.337+0000 7fe0aaf6a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:21:57.336 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:21:57.341+0000 7fe0aaf6a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:21:57.338 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:21:57.341+0000 7fe0aaf6a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:21:57.622 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-25T23:21:57.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:21:57.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T23:21:57.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:21:57.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:21:57.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:21:57.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:21:57.626 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-25T23:21:57.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:21:57.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:21:57.881 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:21:57.885+0000 7fe0aaf6a7c0 -1 Falling back to public interface 2024-03-25T23:21:57.975 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:21:58.976 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:21:58.976 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:21:58.976 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:21:58.976 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:21:58.976 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:21:58.977 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:21:59.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:22:00.300 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:22:00.300 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:22:00.300 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:22:00.300 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:22:00.301 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:22:00.301 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:22:00.628 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:22:00.680 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:22:00.685+0000 7fe0aaf6a7c0 -1 osd.1 26 log_to_monitors true 2024-03-25T23:22:01.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:22:01.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:22:01.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:22:01.631 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:22:01.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:22:01.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:22:01.955 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1 up in weight 1 up_from 30 up_thru 30 down_at 27 last_clean_interval [10,26) [v2:127.0.0.1:6810/2306659864,v1:127.0.0.1:6811/2306659864] [v2:127.0.0.1:6812/2306659864,v1:127.0.0.1:6813/2306659864] exists,up 3511f3d0-5053-4ee7-a03a-3c7ebf69bf89 2024-03-25T23:22:01.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:22:01.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:22:01.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:22:01.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:22:01.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:22:01.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:22:01.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:22:01.956 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:22:01.957 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:22:01.957 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:22:01.957 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:22:01.957 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:22:01.957 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:22:02.066 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:22:02.066 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:22:02.066 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:22:02.066 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:22:02.066 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:22:02.066 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:22:02.380 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:22:02.380 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:22:02.380 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-25T23:22:02.380 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:22:02.381 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:22:02.381 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:22:02.559 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182405 2024-03-25T23:22:02.559 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182405 2024-03-25T23:22:02.559 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405' 2024-03-25T23:22:02.559 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:22:02.560 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:22:02.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-03-25T23:22:02.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-03-25T23:22:02.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405 1-128849018882' 2024-03-25T23:22:02.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:22:02.716 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:22:02.857 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509454 2024-03-25T23:22:02.857 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509454 2024-03-25T23:22:02.857 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405 1-128849018882 2-64424509454' 2024-03-25T23:22:02.857 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:22:02.858 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182405 2024-03-25T23:22:02.858 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:22:02.859 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:22:02.859 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182405 2024-03-25T23:22:02.860 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:22:02.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182405 2024-03-25T23:22:02.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182405' 2024-03-25T23:22:02.861 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 107374182405 2024-03-25T23:22:02.861 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:22:03.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182404 -lt 107374182405 2024-03-25T23:22:03.163 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:22:04.164 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:22:04.165 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:22:04.476 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182405 -lt 107374182405 2024-03-25T23:22:04.476 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:22:04.476 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:22:04.477 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-128849018882 2024-03-25T23:22:04.478 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:22:04.478 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-128849018882 2024-03-25T23:22:04.479 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:22:04.480 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-03-25T23:22:04.480 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 128849018882' 2024-03-25T23:22:04.480 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 128849018882 2024-03-25T23:22:04.480 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:22:04.815 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-03-25T23:22:04.815 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:22:04.815 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509454 2024-03-25T23:22:04.815 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:22:04.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:22:04.817 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509454 2024-03-25T23:22:04.818 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:22:04.819 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509454 2024-03-25T23:22:04.819 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-03-25T23:22:04.819 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 64424509454 2024-03-25T23:22:04.819 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:22:05.131 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509454 -lt 64424509454 2024-03-25T23:22:05.131 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:22:05.131 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:22:05.132 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:22:05.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-25T23:22:05.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:22:05.508 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:22:05.508 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:22:05.508 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:22:05.508 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:22:05.508 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:22:05.508 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:22:05.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-25T23:22:05.810 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:22:05.810 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:22:05.810 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:22:06.185 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-25T23:22:06.185 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:22:06.185 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:22:06.185 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:635: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2024-03-25T23:22:06.186 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-25T23:22:06.186 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-25T23:22:06.186 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-25T23:22:06.187 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-25T23:22:06.187 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-25T23:22:06.187 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-25T23:22:06.187 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-25T23:22:06.504 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-25T23:22:06.505 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-25T23:22:06.505 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-25T23:22:06.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:635: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2024-03-25T23:22:06.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:22:06.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:22:06.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-25T23:22:06.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:22:06.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2024-03-25T23:22:06.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:22:06.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:22:06.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-25T23:22:06.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:22:06.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-25T23:22:06.839 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:22:06.839 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:22:06.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:22:06.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:22:06.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:22:07.144 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:22:07.144 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-03-25T23:22:07.144 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:22:07.144 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:22:07.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-25T23:22:07.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:22:07.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-25T23:22:07.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-03-25T23:22:08.486 INFO:tasks.workunit.client.0.smithi129.stdout:remove #1:104778fc:::obj2:head# 2024-03-25T23:22:09.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:22:09.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:22:09.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:22:09.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-25T23:22:09.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:22:09.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-25T23:22:09.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:22:09.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:22:09.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:22:09.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:22:09.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-25T23:22:09.035 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-25T23:22:09.035 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:22:09.035 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:22:09.035 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:22:09.035 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:22:09.035 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:22:09.035 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:22:09.035 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:22:09.035 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:22:09.035 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:22:09.036 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:22:09.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:22:09.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:22:09.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:22:09.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:22:09.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:22:09.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:22:09.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:22:09.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:22:09.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:22:09.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-25T23:22:09.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-25T23:22:09.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-25T23:22:09.038 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.0 2024-03-25T23:22:09.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:22:09.038 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-25T23:22:09.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-25T23:22:09.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:22:09.039 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:22:09.039 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:22:09.076 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:22:09.080+0000 7f3faf4957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:22:09.079 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:22:09.080+0000 7f3faf4957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:22:09.080 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:22:09.084+0000 7f3faf4957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:22:09.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-25T23:22:09.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:22:09.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T23:22:09.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:22:09.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:22:09.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:22:09.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:22:09.345 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-25T23:22:09.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:22:09.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:22:09.614 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:22:09.616+0000 7f3faf4957c0 -1 Falling back to public interface 2024-03-25T23:22:09.696 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:22:10.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:22:10.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:22:10.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:22:10.699 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:22:10.699 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:22:10.699 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:22:11.014 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:22:12.016 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:22:12.016 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:22:12.016 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:22:12.016 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:22:12.016 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:22:12.016 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:22:12.356 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:22:12.696 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:22:12.696+0000 7f3faf4957c0 -1 osd.0 31 log_to_monitors true 2024-03-25T23:22:13.358 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:22:13.358 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:22:13.358 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:22:13.358 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:22:13.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:22:13.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:22:13.679 INFO:tasks.workunit.client.0.smithi129.stderr:osd.0 up in weight 1 up_from 35 up_thru 27 down_at 32 last_clean_interval [25,31) [v2:127.0.0.1:6802/1554726859,v1:127.0.0.1:6803/1554726859] [v2:127.0.0.1:6804/1554726859,v1:127.0.0.1:6805/1554726859] exists,up 216868d9-b284-4562-9481-c9eb9f7570c1 2024-03-25T23:22:13.679 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:22:13.679 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:22:13.679 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:22:13.679 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:22:13.679 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:22:13.679 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:22:13.679 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:22:13.680 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:22:13.680 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:22:13.680 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:22:13.681 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:22:13.681 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:22:13.681 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:22:13.794 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:22:13.794 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:22:13.794 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:22:13.794 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:22:13.795 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:22:13.795 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:22:14.144 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:22:14.144 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:22:14.144 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-25T23:22:14.144 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:22:14.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:22:14.145 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:22:14.291 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-03-25T23:22:14.291 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-03-25T23:22:14.291 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362' 2024-03-25T23:22:14.292 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:22:14.292 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:22:14.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018885 2024-03-25T23:22:14.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018885 2024-03-25T23:22:14.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-128849018885' 2024-03-25T23:22:14.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:22:14.485 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:22:14.643 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509457 2024-03-25T23:22:14.644 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509457 2024-03-25T23:22:14.644 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-128849018885 2-64424509457' 2024-03-25T23:22:14.644 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:22:14.644 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855362 2024-03-25T23:22:14.644 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:22:14.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:22:14.645 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855362 2024-03-25T23:22:14.646 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:22:14.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-03-25T23:22:14.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2024-03-25T23:22:14.646 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 150323855362 2024-03-25T23:22:14.647 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:22:14.936 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 150323855362 2024-03-25T23:22:14.936 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:22:15.937 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:22:15.938 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:22:16.295 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-03-25T23:22:16.295 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:22:16.296 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-128849018885 2024-03-25T23:22:16.296 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:22:16.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:22:16.298 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-128849018885 2024-03-25T23:22:16.299 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:22:16.300 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018885 2024-03-25T23:22:16.300 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 128849018885' 2024-03-25T23:22:16.300 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 128849018885 2024-03-25T23:22:16.300 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:22:16.639 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018886 -lt 128849018885 2024-03-25T23:22:16.640 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:22:16.640 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509457 2024-03-25T23:22:16.640 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:22:16.642 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:22:16.643 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509457 2024-03-25T23:22:16.643 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:22:16.644 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509457 2024-03-25T23:22:16.644 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-03-25T23:22:16.644 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 64424509457 2024-03-25T23:22:16.644 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:22:16.997 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509458 -lt 64424509457 2024-03-25T23:22:16.998 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:22:16.998 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:22:16.998 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:22:17.396 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-25T23:22:17.396 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:22:17.397 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:22:17.397 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:22:17.397 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:22:17.397 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:22:17.397 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:22:17.397 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:22:17.723 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-25T23:22:17.724 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:22:17.724 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:22:17.724 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:22:18.105 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-25T23:22:18.106 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:22:18.106 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:22:18.106 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:636: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2024-03-25T23:22:18.107 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-25T23:22:18.107 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-25T23:22:18.107 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-25T23:22:18.107 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-25T23:22:18.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:636: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2024-03-25T23:22:18.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:22:18.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:22:18.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-25T23:22:18.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:22:18.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2024-03-25T23:22:18.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:22:18.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:22:18.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-25T23:22:18.432 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:22:18.432 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-25T23:22:18.432 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:22:18.432 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:22:18.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:22:18.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:22:18.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:22:18.741 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:22:18.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2024-03-25T23:22:18.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:22:18.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:22:18.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-25T23:22:18.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:22:18.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:22:18.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2024-03-25T23:22:20.209 INFO:tasks.workunit.client.0.smithi129.stderr:Error getting attr on : 1.0_head,#1:ff7b1f36:::obj1:head#, (61) No data available 2024-03-25T23:22:20.759 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:22:20.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:22:20.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:22:20.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-25T23:22:20.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:22:20.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:22:20.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:22:20.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:22:20.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:22:20.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:22:20.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-25T23:22:20.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-25T23:22:20.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:22:20.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:22:20.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:22:20.761 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:22:20.761 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:22:20.761 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:22:20.761 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:22:20.761 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:22:20.761 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:22:20.761 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:22:20.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:22:20.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:22:20.762 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:22:20.762 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:22:20.762 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:22:20.762 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:22:20.762 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:22:20.762 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:22:20.762 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:22:20.762 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-25T23:22:20.762 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-25T23:22:20.763 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-25T23:22:20.763 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.1 2024-03-25T23:22:20.763 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:22:20.763 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-25T23:22:20.764 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-25T23:22:20.764 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:22:20.764 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:22:20.764 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:22:20.797 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:22:20.800+0000 7fa72a8577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:22:20.799 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:22:20.804+0000 7fa72a8577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:22:20.801 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:22:20.804+0000 7fa72a8577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:22:21.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-25T23:22:21.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:22:21.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T23:22:21.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:22:21.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:22:21.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:22:21.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:22:21.086 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-25T23:22:21.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:22:21.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:22:21.354 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:22:21.356+0000 7fa72a8577c0 -1 Falling back to public interface 2024-03-25T23:22:21.402 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:22:22.402 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:22:22.402 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:22:22.403 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:22:22.403 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:22:22.403 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:22:22.403 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:22:22.714 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:22:23.715 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:22:23.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:22:23.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:22:23.716 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:22:23.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:22:23.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:22:23.965 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:22:23.968+0000 7fa72a8577c0 -1 osd.1 36 log_to_monitors true 2024-03-25T23:22:24.025 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:22:25.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:22:25.028 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:22:25.028 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:22:25.028 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:22:25.028 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:22:25.028 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:22:25.457 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1 up in weight 1 up_from 40 up_thru 40 down_at 37 last_clean_interval [30,36) [v2:127.0.0.1:6810/2850662604,v1:127.0.0.1:6811/2850662604] [v2:127.0.0.1:6812/2850662604,v1:127.0.0.1:6813/2850662604] exists,up 3511f3d0-5053-4ee7-a03a-3c7ebf69bf89 2024-03-25T23:22:25.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:22:25.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:22:25.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:22:25.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:22:25.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:22:25.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:22:25.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:22:25.458 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:22:25.458 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:22:25.458 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:22:25.458 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:22:25.459 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:22:25.459 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:22:25.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:22:25.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:22:25.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:22:25.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:22:25.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:22:25.562 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:22:25.881 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:22:25.881 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:22:25.881 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-25T23:22:25.881 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:22:25.881 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:22:25.881 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:22:26.056 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855365 2024-03-25T23:22:26.056 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855365 2024-03-25T23:22:26.056 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855365' 2024-03-25T23:22:26.056 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:22:26.056 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:22:26.196 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-03-25T23:22:26.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-03-25T23:22:26.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855365 1-171798691842' 2024-03-25T23:22:26.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:22:26.197 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:22:26.340 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509461 2024-03-25T23:22:26.340 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509461 2024-03-25T23:22:26.340 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855365 1-171798691842 2-64424509461' 2024-03-25T23:22:26.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:22:26.341 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:22:26.341 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855365 2024-03-25T23:22:26.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:22:26.342 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855365 2024-03-25T23:22:26.342 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:22:26.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855365 2024-03-25T23:22:26.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855365' 2024-03-25T23:22:26.343 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 150323855365 2024-03-25T23:22:26.344 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:22:26.638 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855364 -lt 150323855365 2024-03-25T23:22:26.638 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:22:27.639 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:22:27.640 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:22:27.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855364 -lt 150323855365 2024-03-25T23:22:27.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:22:28.950 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-25T23:22:28.951 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:22:29.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855365 -lt 150323855365 2024-03-25T23:22:29.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:22:29.264 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691842 2024-03-25T23:22:29.264 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:22:29.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:22:29.266 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:22:29.266 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691842 2024-03-25T23:22:29.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-03-25T23:22:29.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691842' 2024-03-25T23:22:29.267 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 171798691842 2024-03-25T23:22:29.268 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:22:29.593 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691842 -lt 171798691842 2024-03-25T23:22:29.593 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:22:29.594 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509461 2024-03-25T23:22:29.594 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:22:29.595 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:22:29.596 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509461 2024-03-25T23:22:29.596 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:22:29.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509461 2024-03-25T23:22:29.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-03-25T23:22:29.598 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 64424509461 2024-03-25T23:22:29.599 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:22:29.939 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509461 -lt 64424509461 2024-03-25T23:22:29.940 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:22:29.941 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:22:29.941 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:22:30.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-25T23:22:30.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:22:30.383 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:22:30.383 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:22:30.383 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:22:30.383 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:22:30.383 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:22:30.383 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:22:30.700 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-25T23:22:30.700 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:22:30.701 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:22:30.701 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:22:31.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-25T23:22:31.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:22:31.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:22:31.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:637: TEST_auto_repair_bluestore_failed_norecov: ceph tell 'osd.*' config set osd_scrub_auto_repair true 2024-03-25T23:22:31.225 INFO:tasks.workunit.client.0.smithi129.stdout:osd.0: { 2024-03-25T23:22:31.225 INFO:tasks.workunit.client.0.smithi129.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-03-25T23:22:31.225 INFO:tasks.workunit.client.0.smithi129.stdout:} 2024-03-25T23:22:31.238 INFO:tasks.workunit.client.0.smithi129.stdout:osd.1: { 2024-03-25T23:22:31.238 INFO:tasks.workunit.client.0.smithi129.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-03-25T23:22:31.238 INFO:tasks.workunit.client.0.smithi129.stdout:} 2024-03-25T23:22:31.253 INFO:tasks.workunit.client.0.smithi129.stdout:osd.2: { 2024-03-25T23:22:31.253 INFO:tasks.workunit.client.0.smithi129.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-03-25T23:22:31.253 INFO:tasks.workunit.client.0.smithi129.stdout:} 2024-03-25T23:22:31.265 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:639: TEST_auto_repair_bluestore_failed_norecov: get_pg testpool obj1 2024-03-25T23:22:31.265 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-03-25T23:22:31.265 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-03-25T23:22:31.265 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-03-25T23:22:31.265 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-25T23:22:31.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:639: TEST_auto_repair_bluestore_failed_norecov: local pgid=1.0 2024-03-25T23:22:31.581 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:640: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool obj1 2024-03-25T23:22:31.582 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-25T23:22:31.582 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-03-25T23:22:31.582 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-03-25T23:22:31.582 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-25T23:22:31.875 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:640: TEST_auto_repair_bluestore_failed_norecov: local primary=1 2024-03-25T23:22:31.875 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:641: TEST_auto_repair_bluestore_failed_norecov: get_last_scrub_stamp 1.0 2024-03-25T23:22:31.875 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-25T23:22:31.875 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:22:31.875 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:22:31.876 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-25T23:22:32.196 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:641: TEST_auto_repair_bluestore_failed_norecov: local last_scrub_stamp=2024-03-25T23:21:30.918519+0000 2024-03-25T23:22:32.196 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:642: TEST_auto_repair_bluestore_failed_norecov: ceph tell 1.0 schedule-deep-scrub 2024-03-25T23:22:32.331 INFO:tasks.workunit.client.0.smithi129.stdout:{ 2024-03-25T23:22:32.331 INFO:tasks.workunit.client.0.smithi129.stdout: "deep": true, 2024-03-25T23:22:32.331 INFO:tasks.workunit.client.0.smithi129.stdout: "must": false, 2024-03-25T23:22:32.331 INFO:tasks.workunit.client.0.smithi129.stdout: "stamp": "2024-03-11T23:20:52.336967+0000" 2024-03-25T23:22:32.331 INFO:tasks.workunit.client.0.smithi129.stdout:} 2024-03-25T23:22:32.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:645: TEST_auto_repair_bluestore_failed_norecov: wait_for_scrub 1.0 2024-03-25T23:21:30.918519+0000 2024-03-25T23:22:32.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-03-25T23:22:32.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-25T23:21:30.918519+0000 2024-03-25T23:22:32.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-25T23:22:32.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-25T23:22:32.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:22:32.346 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-25T23:22:32.346 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-25T23:22:32.346 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:22:32.346 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:22:32.346 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-25T23:22:32.652 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:21:30.918519+0000 '>' 2024-03-25T23:21:30.918519+0000 2024-03-25T23:22:32.652 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:22:33.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:22:33.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:22:33.655 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-25T23:22:33.655 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-25T23:22:33.655 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:22:33.656 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:22:33.656 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-25T23:22:33.954 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:21:30.918519+0000 '>' 2024-03-25T23:21:30.918519+0000 2024-03-25T23:22:33.954 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:22:34.955 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:22:34.955 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:22:34.956 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-25T23:22:34.956 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-25T23:22:34.956 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:22:34.956 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:22:34.957 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-25T23:22:35.282 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:21:30.918519+0000 '>' 2024-03-25T23:21:30.918519+0000 2024-03-25T23:22:35.282 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:22:36.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:22:36.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:22:36.284 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-25T23:22:36.284 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-25T23:22:36.284 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:22:36.285 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:22:36.285 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-25T23:22:36.590 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-11T23:20:52.336967+0000 '>' 2024-03-25T23:21:30.918519+0000 2024-03-25T23:22:36.590 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:22:37.591 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:22:37.591 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:22:37.592 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-25T23:22:37.592 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-25T23:22:37.592 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:22:37.593 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:22:37.593 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-25T23:22:37.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-11T23:20:52.336967+0000 '>' 2024-03-25T23:21:30.918519+0000 2024-03-25T23:22:37.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:22:38.909 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:22:38.909 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:22:38.910 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-25T23:22:38.910 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-25T23:22:38.910 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:22:38.911 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:22:38.911 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-25T23:22:39.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-11T23:20:52.336967+0000 '>' 2024-03-25T23:21:30.918519+0000 2024-03-25T23:22:39.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:22:40.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:22:40.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:22:40.233 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-25T23:22:40.233 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-25T23:22:40.233 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:22:40.233 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:22:40.234 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-25T23:22:40.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:22:34.995438+0000 '>' 2024-03-25T23:21:30.918519+0000 2024-03-25T23:22:40.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-25T23:22:40.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:646: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2024-03-25T23:22:40.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:22:40.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:22:40.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:22:40.554 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:22:40.554 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:22:40.554 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:22:40.554 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:22:40.555 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:22:40.555 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:22:40.662 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:22:40.662 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:22:40.662 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:22:40.662 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:22:40.662 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:22:40.662 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:22:40.984 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:22:40.984 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:22:40.984 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-25T23:22:40.984 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:22:40.984 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:22:40.984 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:22:41.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855369 2024-03-25T23:22:41.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855369 2024-03-25T23:22:41.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855369' 2024-03-25T23:22:41.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:22:41.153 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:22:41.294 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691846 2024-03-25T23:22:41.294 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691846 2024-03-25T23:22:41.294 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855369 1-171798691846' 2024-03-25T23:22:41.294 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:22:41.294 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:22:41.436 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509465 2024-03-25T23:22:41.437 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509465 2024-03-25T23:22:41.437 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855369 1-171798691846 2-64424509465' 2024-03-25T23:22:41.437 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:22:41.437 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855369 2024-03-25T23:22:41.437 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:22:41.439 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:22:41.439 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855369 2024-03-25T23:22:41.439 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:22:41.441 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855369 2024-03-25T23:22:41.441 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 150323855369 2024-03-25T23:22:41.441 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855369' 2024-03-25T23:22:41.441 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:22:41.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855368 -lt 150323855369 2024-03-25T23:22:41.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:22:42.744 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:22:42.744 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:22:43.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855369 -lt 150323855369 2024-03-25T23:22:43.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:22:43.047 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:22:43.047 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691846 2024-03-25T23:22:43.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:22:43.049 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691846 2024-03-25T23:22:43.049 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:22:43.050 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691846 2024-03-25T23:22:43.051 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 171798691846 2024-03-25T23:22:43.051 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691846' 2024-03-25T23:22:43.051 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:22:43.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691846 -lt 171798691846 2024-03-25T23:22:43.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:22:43.391 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509465 2024-03-25T23:22:43.391 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:22:43.393 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:22:43.393 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509465 2024-03-25T23:22:43.394 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:22:43.395 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509465 2024-03-25T23:22:43.395 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 64424509465 2024-03-25T23:22:43.395 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509465' 2024-03-25T23:22:43.395 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:22:43.714 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509465 -lt 64424509465 2024-03-25T23:22:43.714 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:22:43.715 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:22:43.715 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:22:44.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-25T23:22:44.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:22:44.124 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:22:44.124 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:22:44.124 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:22:44.124 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:22:44.124 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:22:44.124 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:22:44.441 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-25T23:22:44.441 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:22:44.442 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:22:44.442 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:22:44.798 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-25T23:22:44.798 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:22:44.798 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:22:44.798 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:647: TEST_auto_repair_bluestore_failed_norecov: flush_pg_stats 2024-03-25T23:22:44.798 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:22:44.799 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:22:45.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:22:45.166 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:22:45.166 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-25T23:22:45.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:22:45.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:22:45.166 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:22:45.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855371 2024-03-25T23:22:45.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855371 2024-03-25T23:22:45.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855371' 2024-03-25T23:22:45.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:22:45.341 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:22:45.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691848 2024-03-25T23:22:45.482 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691848 2024-03-25T23:22:45.482 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855371 1-171798691848' 2024-03-25T23:22:45.482 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:22:45.482 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:22:45.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509467 2024-03-25T23:22:45.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509467 2024-03-25T23:22:45.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855371 1-171798691848 2-64424509467' 2024-03-25T23:22:45.624 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:22:45.624 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855371 2024-03-25T23:22:45.624 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:22:45.625 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:22:45.625 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855371 2024-03-25T23:22:45.625 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:22:45.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855371 2024-03-25T23:22:45.627 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 150323855371 2024-03-25T23:22:45.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855371' 2024-03-25T23:22:45.627 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:22:45.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855370 -lt 150323855371 2024-03-25T23:22:45.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:22:46.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:22:46.929 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:22:47.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855371 -lt 150323855371 2024-03-25T23:22:47.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:22:47.232 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:22:47.233 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691848 2024-03-25T23:22:47.234 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:22:47.234 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691848 2024-03-25T23:22:47.235 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:22:47.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691848 2024-03-25T23:22:47.236 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 171798691848 2024-03-25T23:22:47.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691848' 2024-03-25T23:22:47.236 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:22:47.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691848 -lt 171798691848 2024-03-25T23:22:47.573 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:22:47.574 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509467 2024-03-25T23:22:47.574 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:22:47.575 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:22:47.576 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509467 2024-03-25T23:22:47.576 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:22:47.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509467 2024-03-25T23:22:47.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509467' 2024-03-25T23:22:47.578 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 64424509467 2024-03-25T23:22:47.578 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:22:47.924 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509467 -lt 64424509467 2024-03-25T23:22:47.924 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:648: TEST_auto_repair_bluestore_failed_norecov: grep -q 'scrub_finish.*present with no repair possible' td/osd-scrub-repair/osd.1.log 2024-03-25T23:22:47.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:649: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2024-03-25T23:22:48.228 INFO:tasks.workunit.client.0.smithi129.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-25T23:22:48.228 INFO:tasks.workunit.client.0.smithi129.stdout:1.0 10 0 0 0 0 56 0 0 10 0 10 active+clean+inconsistent+failed_repair 2024-03-25T23:22:34.995723+0000 21'10 41:93 [1,0] 1 [1,0] 1 21'10 2024-03-25T23:22:34.995438+0000 21'10 2024-03-25T23:22:34.995438+0000 0 1 periodic scrub scheduled @ 2024-03-26T23:22:34.995438+0000 10 0 2024-03-25T23:22:48.228 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:22:48.228 INFO:tasks.workunit.client.0.smithi129.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-25T23:22:48.228 INFO:tasks.workunit.client.0.smithi129.stderr:dumped pgs 2024-03-25T23:22:48.244 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:650: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2024-03-25T23:22:48.244 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:650: TEST_auto_repair_bluestore_failed_norecov: grep -q '^1.0.*+failed_repair' 2024-03-25T23:22:48.535 INFO:tasks.workunit.client.0.smithi129.stderr:dumped pgs 2024-03-25T23:22:48.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-25T23:22:48.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-25T23:22:48.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T23:22:48.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-25T23:22:48.553 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:22:48.553 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:22:48.554 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:22:48.554 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:22:48.554 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:22:48.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:22:48.665 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T23:22:48.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T23:22:48.666 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T23:22:48.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T23:22:48.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T23:22:48.667 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T23:22:48.668 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:22:48.668 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T23:22:48.668 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T23:22:48.668 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:22:48.669 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T23:22:48.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T23:22:48.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-25T23:22:48.688 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T23:22:48.688 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:22:48.688 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:22:48.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-25T23:22:48.690 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T23:22:48.690 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T23:22:48.691 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-25T23:22:48.691 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-25T23:22:48.691 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-25T23:22:48.691 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-25T23:22:48.691 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-25T23:22:48.691 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T23:22:48.691 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-25T23:22:48.691 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:22:48.691 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:22:48.691 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:22:48.691 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:22:48.691 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:22:48.692 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:22:48.692 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T23:22:48.693 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T23:22:48.693 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T23:22:48.694 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T23:22:48.694 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T23:22:48.695 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T23:22:48.695 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:22:48.695 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T23:22:48.696 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T23:22:48.696 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:22:48.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T23:22:48.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T23:22:48.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-25T23:22:48.699 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T23:22:48.699 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:22:48.699 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:22:48.699 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-25T23:22:48.700 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T23:22:48.700 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T23:22:48.700 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-25T23:22:48.701 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-25T23:22:48.701 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:22:48.701 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:22:48.701 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61636 2024-03-25T23:22:48.703 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-25T23:22:48.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-25T23:22:48.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-25T23:22:48.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-25T23:22:48.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_scrub td/osd-scrub-repair 2024-03-25T23:22:48.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:483: TEST_auto_repair_bluestore_scrub: local dir=td/osd-scrub-repair 2024-03-25T23:22:48.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:484: TEST_auto_repair_bluestore_scrub: local poolname=testpool 2024-03-25T23:22:48.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:487: TEST_auto_repair_bluestore_scrub: run_mon td/osd-scrub-repair a 2024-03-25T23:22:48.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-25T23:22:48.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-25T23:22:48.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-25T23:22:48.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-25T23:22:48.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-25T23:22:48.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-25T23:22:49.161 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-25T23:22:49.161 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:22:49.161 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:22:49.162 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:22:49.162 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:22:49.162 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:22:49.162 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:22:49.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-25T23:22:49.356 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-25T23:22:49.359 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-25T23:22:49.359 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T23:22:49.359 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T23:22:49.360 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-25T23:22:49.360 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-25T23:22:49.360 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T23:22:49.360 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T23:22:49.360 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T23:22:49.360 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T23:22:49.361 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:22:49.361 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:22:49.361 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-25T23:22:49.361 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T23:22:49.361 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get fsid 2024-03-25T23:22:49.476 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-25T23:22:49.476 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T23:22:49.476 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T23:22:49.476 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-25T23:22:49.477 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T23:22:49.477 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-25T23:22:49.477 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T23:22:49.478 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T23:22:49.478 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T23:22:49.478 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:22:49.478 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:22:49.478 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-25T23:22:49.478 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T23:22:49.478 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get mon_host 2024-03-25T23:22:49.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:488: TEST_auto_repair_bluestore_scrub: run_mgr td/osd-scrub-repair x 2024-03-25T23:22:49.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-25T23:22:49.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-25T23:22:49.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-25T23:22:49.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-25T23:22:49.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-25T23:22:49.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-25T23:22:50.466 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-25T23:22:50.466 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:22:50.466 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:22:50.467 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:22:50.467 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:22:50.467 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:22:50.467 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:22:50.468 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T23:22:50.469 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T23:22:50.503 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:492: TEST_auto_repair_bluestore_scrub: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-03-25T23:22:50.503 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: seq 0 2 2024-03-25T23:22:50.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-03-25T23:22:50.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-25T23:22:50.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:22:50.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:22:50.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-25T23:22:50.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:22:50.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-25T23:22:50.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:22:50.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:22:50.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:22:50.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:22:50.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-25T23:22:50.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-25T23:22:50.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:22:50.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:22:50.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:22:50.508 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:22:50.508 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:22:50.508 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:22:50.508 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:22:50.508 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:22:50.509 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:22:50.509 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:22:50.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:22:50.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:22:50.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:22:50.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:22:50.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:22:50.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:22:50.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:22:50.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:22:50.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:22:50.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:22:50.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:22:50.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-03-25T23:22:50.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-25T23:22:50.510 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:22:50.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9e4ff428-8798-4568-a998-3f2b8861361b 2024-03-25T23:22:50.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9e4ff428-8798-4568-a998-3f2b8861361b' 2024-03-25T23:22:50.511 INFO:tasks.workunit.client.0.smithi129.stdout:add osd0 9e4ff428-8798-4568-a998-3f2b8861361b 2024-03-25T23:22:50.511 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:22:50.526 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBKBwJm7oasHxAA98gnZsdeWbVw6ZsS2C5JFw== 2024-03-25T23:22:50.526 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBKBwJm7oasHxAA98gnZsdeWbVw6ZsS2C5JFw=="}' 2024-03-25T23:22:50.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9e4ff428-8798-4568-a998-3f2b8861361b -i td/osd-scrub-repair/0/new.json 2024-03-25T23:22:50.792 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:22:50.808 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-25T23:22:50.809 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQBKBwJm7oasHxAA98gnZsdeWbVw6ZsS2C5JFw== --osd-uuid 9e4ff428-8798-4568-a998-3f2b8861361b 2024-03-25T23:22:50.844 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:22:50.847+0000 7f8608d007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:22:50.859 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:22:50.859+0000 7f8608d007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:22:50.861 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:22:50.863+0000 7f8608d007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:22:50.861 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:22:50.863+0000 7f8608d007c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-25T23:22:50.861 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:22:50.863+0000 7f8608d007c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-25T23:22:50.861 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:22:50.863+0000 7f8608d007c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-25T23:22:50.960 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:22:50.963+0000 7f8608d007c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-25T23:22:57.268 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-25T23:22:57.268 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:22:57.269 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-25T23:22:57.269 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd0 key to auth repository 2024-03-25T23:22:57.270 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:22:57.654 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.0 2024-03-25T23:22:57.655 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-25T23:22:57.655 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-25T23:22:57.655 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:22:57.655 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:22:57.655 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:22:57.687 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:22:57.687+0000 7fdd1a73d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:22:57.718 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:22:57.719+0000 7fdd1a73d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:22:57.719 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:22:57.719+0000 7fdd1a73d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:22:57.985 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-25T23:22:57.985 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:22:57.985 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T23:22:57.985 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:22:57.985 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:22:57.985 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:22:57.985 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:22:57.986 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:22:57.986 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:22:57.986 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:22:58.279 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:22:58.279+0000 7fdd1a73d7c0 -1 Falling back to public interface 2024-03-25T23:22:58.327 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:22:59.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:22:59.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:22:59.329 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:22:59.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:22:59.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:22:59.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:22:59.639 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:23:00.641 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:23:00.641 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:23:00.641 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:23:00.642 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:23:00.642 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:23:00.642 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:23:00.856 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:23:00.855+0000 7fdd1a73d7c0 -1 osd.0 0 log_to_monitors true 2024-03-25T23:23:00.974 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:23:01.976 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:23:01.977 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:23:01.977 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:23:01.978 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:23:01.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:23:01.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:23:02.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:23:03.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:23:03.298 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:23:03.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:23:03.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:23:03.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:23:03.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:23:03.775 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:23:04.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:23:04.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:23:04.777 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:23:04.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:23:04.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:23:04.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:23:05.100 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:23:06.101 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:23:06.101 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:23:06.101 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:23:06.101 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:23:06.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:23:06.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:23:06.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:23:07.422 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:23:07.422 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:23:07.423 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:23:07.423 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:23:07.423 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:23:07.424 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:23:07.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:23:08.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:23:08.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:23:08.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:23:08.735 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:23:08.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:23:08.736 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:23:09.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:23:10.044 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:23:10.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:23:10.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:23:10.045 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:23:10.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:23:10.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:23:10.356 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:23:11.357 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:23:11.358 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:23:11.358 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:23:11.358 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:23:11.358 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:23:11.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:23:11.709 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:23:12.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:23:12.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:23:12.711 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:23:12.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:23:12.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:23:12.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:23:13.026 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:23:14.028 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:23:14.028 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:23:14.028 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:23:14.029 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:23:14.029 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:23:14.029 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:23:14.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:23:15.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:23:15.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:23:15.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:23:15.347 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:23:15.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:23:15.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:23:15.883 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:23:16.885 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:23:16.885 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:23:16.885 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:23:16.886 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:23:16.886 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:23:16.886 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:23:17.763 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:23:18.766 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:23:18.766 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:23:18.766 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:23:18.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:23:18.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:23:18.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:23:19.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:23:20.080 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:23:20.080 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:23:20.080 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:23:20.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:23:20.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:23:20.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:23:20.793 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:23:21.795 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:23:21.795 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:23:21.795 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:23:21.796 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:23:21.796 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:23:21.796 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:23:22.663 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:23:23.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:23:23.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:23:23.665 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:23:23.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:23:23.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:23:23.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:23:24.185 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:23:25.187 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:23:25.187 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:23:25.187 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:23:25.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:23:25.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:23:25.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:23:25.516 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:23:26.518 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:23:26.518 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:23:26.518 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-25T23:23:26.519 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-25T23:23:26.519 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:23:26.519 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:23:26.837 INFO:tasks.workunit.client.0.smithi129.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1494065369,v1:127.0.0.1:6803/1494065369] [v2:127.0.0.1:6804/1494065369,v1:127.0.0.1:6805/1494065369] exists,up 9e4ff428-8798-4568-a998-3f2b8861361b 2024-03-25T23:23:26.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:23:26.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:23:26.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:23:26.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-03-25T23:23:26.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-25T23:23:26.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:23:26.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:23:26.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-25T23:23:26.841 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:23:26.841 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:23:26.841 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:23:26.841 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:23:26.841 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:23:26.841 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:23:26.841 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-25T23:23:26.841 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-25T23:23:26.841 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:23:26.841 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:23:26.841 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:23:26.841 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:23:26.841 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:23:26.841 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:23:26.842 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:23:26.842 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:23:26.842 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:23:26.842 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:23:26.843 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:23:26.843 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:23:26.843 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:23:26.844 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:23:26.844 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:23:26.844 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:23:26.844 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:23:26.844 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:23:26.844 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:23:26.844 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:23:26.844 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:23:26.844 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-03-25T23:23:26.844 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-25T23:23:26.844 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:23:26.845 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1729d8f9-6ae2-422b-8c65-9d3ad8eaf1a7 2024-03-25T23:23:26.845 INFO:tasks.workunit.client.0.smithi129.stdout:add osd1 1729d8f9-6ae2-422b-8c65-9d3ad8eaf1a7 2024-03-25T23:23:26.845 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 1729d8f9-6ae2-422b-8c65-9d3ad8eaf1a7' 2024-03-25T23:23:26.845 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:23:26.867 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBuBwJme3P3MxAA+Wn2dg6kcvrhjBBpoUPRfw== 2024-03-25T23:23:26.867 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBuBwJme3P3MxAA+Wn2dg6kcvrhjBBpoUPRfw=="}' 2024-03-25T23:23:26.867 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1729d8f9-6ae2-422b-8c65-9d3ad8eaf1a7 -i td/osd-scrub-repair/1/new.json 2024-03-25T23:23:27.273 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:23:27.293 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-25T23:23:27.296 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQBuBwJme3P3MxAA+Wn2dg6kcvrhjBBpoUPRfw== --osd-uuid 1729d8f9-6ae2-422b-8c65-9d3ad8eaf1a7 2024-03-25T23:23:27.332 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:23:27.334+0000 7f12118997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:23:27.376 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:23:27.374+0000 7f12118997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:23:27.377 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:23:27.378+0000 7f12118997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:23:27.378 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:23:27.378+0000 7f12118997c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-25T23:23:27.378 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:23:27.378+0000 7f12118997c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-25T23:23:27.378 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:23:27.378+0000 7f12118997c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-25T23:23:27.458 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:23:27.458+0000 7f12118997c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-25T23:23:33.893 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-25T23:23:33.894 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:23:33.895 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd1 key to auth repository 2024-03-25T23:23:33.895 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-25T23:23:33.895 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:23:34.271 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.1 2024-03-25T23:23:34.272 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-25T23:23:34.272 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:23:34.272 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-25T23:23:34.272 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:23:34.272 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:23:34.306 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:23:34.306+0000 7f540501e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:23:34.338 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:23:34.338+0000 7f540501e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:23:34.339 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:23:34.338+0000 7f540501e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:23:34.622 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-25T23:23:34.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:23:34.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T23:23:34.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:23:34.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:23:34.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:23:34.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:23:34.623 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:23:34.624 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:23:34.624 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:23:34.903 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:23:34.902+0000 7f540501e7c0 -1 Falling back to public interface 2024-03-25T23:23:34.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:23:35.981 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:23:35.981 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:23:35.981 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:23:35.982 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:23:35.982 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:23:35.982 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:23:36.287 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:23:37.289 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:23:37.289 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:23:37.289 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:23:37.290 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:23:37.290 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:23:37.290 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:23:37.637 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:23:37.920 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:23:37.918+0000 7f540501e7c0 -1 osd.1 0 log_to_monitors true 2024-03-25T23:23:38.639 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:23:38.639 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:23:38.639 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:23:38.640 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:23:38.640 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:23:38.640 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:23:39.063 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:23:40.065 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:23:40.066 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:23:40.066 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:23:40.066 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:23:40.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:23:40.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:23:40.384 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:23:41.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:23:41.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:23:41.387 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:23:41.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:23:41.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:23:41.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:23:41.709 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:23:42.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:23:42.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:23:42.711 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:23:42.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:23:42.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:23:42.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:23:43.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:23:44.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:23:44.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:23:44.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:23:44.034 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:23:44.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:23:44.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:23:44.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:23:45.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:23:45.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:23:45.345 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:23:45.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:23:45.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:23:45.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:23:45.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:23:46.652 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:23:46.652 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:23:46.653 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:23:46.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:23:46.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:23:46.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:23:47.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:23:48.019 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:23:48.020 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:23:48.020 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:23:48.020 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:23:48.020 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:23:48.020 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:23:48.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:23:49.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:23:49.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:23:49.333 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:23:49.333 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:23:49.333 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:23:49.333 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:23:49.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:23:50.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:23:50.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:23:50.655 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:23:50.655 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:23:50.655 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:23:50.655 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:23:51.103 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:23:52.105 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:23:52.105 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:23:52.105 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:23:52.106 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:23:52.106 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:23:52.106 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:23:52.421 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:23:53.423 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:23:53.423 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:23:53.423 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:23:53.424 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:23:53.424 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:23:53.424 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:23:54.285 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:23:55.287 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:23:55.287 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:23:55.287 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:23:55.288 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:23:55.288 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:23:55.288 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:23:55.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:23:56.975 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:23:56.975 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:23:56.976 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:23:56.976 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:23:56.976 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:23:56.976 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:23:57.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:23:58.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:23:58.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:23:58.652 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:23:58.652 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:23:58.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:23:58.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:23:59.306 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:24:00.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:24:00.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:24:00.307 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:24:00.308 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:24:00.308 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:24:00.308 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:24:00.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:24:01.932 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:24:01.932 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:24:01.933 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:24:01.933 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:24:01.933 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:24:01.933 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:24:02.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:24:03.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:24:03.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:24:03.510 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-25T23:24:03.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-25T23:24:03.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:24:03.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:24:03.867 INFO:tasks.workunit.client.0.smithi129.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2907556837,v1:127.0.0.1:6811/2907556837] [v2:127.0.0.1:6812/2907556837,v1:127.0.0.1:6813/2907556837] exists,up 1729d8f9-6ae2-422b-8c65-9d3ad8eaf1a7 2024-03-25T23:24:03.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:24:03.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:24:03.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:24:03.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-03-25T23:24:03.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-25T23:24:03.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:24:03.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:24:03.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-25T23:24:03.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:24:03.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-25T23:24:03.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:24:03.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:24:03.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:24:03.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:24:03.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-25T23:24:03.871 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-25T23:24:03.871 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:24:03.871 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:24:03.871 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:24:03.871 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:24:03.871 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:24:03.871 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:24:03.872 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:24:03.873 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:24:03.873 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:24:03.873 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:24:03.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:24:03.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:24:03.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:24:03.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:24:03.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:24:03.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:24:03.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:24:03.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:24:03.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:24:03.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:24:03.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:24:03.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-03-25T23:24:03.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-25T23:24:03.874 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:24:03.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6e075460-e892-46bb-9233-1017e071bf13 2024-03-25T23:24:03.875 INFO:tasks.workunit.client.0.smithi129.stdout:add osd2 6e075460-e892-46bb-9233-1017e071bf13 2024-03-25T23:24:03.875 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 6e075460-e892-46bb-9233-1017e071bf13' 2024-03-25T23:24:03.875 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:24:03.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCTBwJmNFa6NRAAXbVapHI5EWgUZyq7n0V1zA== 2024-03-25T23:24:03.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCTBwJmNFa6NRAAXbVapHI5EWgUZyq7n0V1zA=="}' 2024-03-25T23:24:03.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6e075460-e892-46bb-9233-1017e071bf13 -i td/osd-scrub-repair/2/new.json 2024-03-25T23:24:04.349 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:24:04.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-25T23:24:04.372 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQCTBwJmNFa6NRAAXbVapHI5EWgUZyq7n0V1zA== --osd-uuid 6e075460-e892-46bb-9233-1017e071bf13 2024-03-25T23:24:04.410 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:24:04.413+0000 7fcd8c7fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:24:04.434 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:24:04.437+0000 7fcd8c7fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:24:04.435 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:24:04.437+0000 7fcd8c7fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:24:04.435 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:24:04.437+0000 7fcd8c7fb7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-25T23:24:04.435 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:24:04.437+0000 7fcd8c7fb7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-25T23:24:04.435 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:24:04.437+0000 7fcd8c7fb7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-25T23:24:04.692 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:24:04.693+0000 7fcd8c7fb7c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-25T23:24:11.365 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-25T23:24:11.365 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:24:11.366 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-25T23:24:11.367 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd2 key to auth repository 2024-03-25T23:24:11.367 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:24:11.757 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.2 2024-03-25T23:24:11.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-25T23:24:11.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:24:11.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:24:11.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-25T23:24:11.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:24:11.793 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:24:11.797+0000 7ff742b267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:24:11.815 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:24:11.817+0000 7ff742b267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:24:11.816 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:24:11.817+0000 7ff742b267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:24:12.076 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-25T23:24:12.076 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:24:12.076 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-25T23:24:12.076 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:24:12.076 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:24:12.077 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:24:12.077 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:24:12.077 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:24:12.077 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:24:12.077 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:24:12.384 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:24:12.386 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:24:12.389+0000 7ff742b267c0 -1 Falling back to public interface 2024-03-25T23:24:13.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:24:13.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:24:13.387 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:24:13.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:24:13.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:24:13.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:24:13.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:24:14.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:24:14.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:24:14.713 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:24:14.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:24:14.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:24:14.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:24:14.962 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:24:14.965+0000 7ff742b267c0 -1 osd.2 0 log_to_monitors true 2024-03-25T23:24:15.082 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:24:16.084 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:24:16.084 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:24:16.084 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:24:16.084 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:24:16.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:24:16.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:24:16.402 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:24:17.404 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:24:17.404 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:24:17.404 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:24:17.405 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:24:17.405 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:24:17.405 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:24:17.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:24:18.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:24:18.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:24:18.832 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:24:18.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:24:18.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:24:18.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:24:19.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:24:20.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:24:20.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:24:20.178 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:24:20.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:24:20.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:24:20.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:24:20.503 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:24:21.502 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:24:21.502 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:24:21.502 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:24:21.503 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:24:21.503 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:24:21.503 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:24:21.858 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:24:22.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:24:22.862 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:24:22.862 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:24:22.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:24:22.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:24:22.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:24:23.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:24:24.180 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:24:24.180 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:24:24.180 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:24:24.181 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:24:24.181 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:24:24.181 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:24:24.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:24:25.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:24:25.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:24:25.497 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:24:25.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:24:25.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:24:25.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:24:25.866 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:24:26.867 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:24:26.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:24:26.868 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:24:26.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:24:26.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:24:26.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:24:27.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:24:28.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:24:28.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:24:28.175 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:24:28.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:24:28.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:24:28.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:24:28.491 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:24:29.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:24:29.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:24:29.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:24:29.493 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:24:29.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:24:29.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:24:30.383 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:24:31.385 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:24:31.385 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:24:31.386 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:24:31.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:24:31.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:24:31.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:24:32.164 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:24:33.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:24:33.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:24:33.167 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:24:33.167 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:24:33.167 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:24:33.167 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:24:33.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:24:34.714 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:24:34.714 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:24:34.715 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:24:34.715 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:24:34.715 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:24:34.715 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:24:35.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:24:36.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:24:36.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:24:36.581 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:24:36.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:24:36.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:24:36.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:24:37.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:24:38.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:24:38.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:24:38.202 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:24:38.202 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:24:38.202 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:24:38.202 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:24:38.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:24:39.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:24:39.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:24:39.837 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:24:39.837 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:24:39.837 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:24:39.837 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:24:40.159 INFO:tasks.workunit.client.0.smithi129.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/711597848,v1:127.0.0.1:6819/711597848] [v2:127.0.0.1:6820/711597848,v1:127.0.0.1:6821/711597848] exists,up 6e075460-e892-46bb-9233-1017e071bf13 2024-03-25T23:24:40.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:24:40.161 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:24:40.161 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:24:40.161 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:497: TEST_auto_repair_bluestore_scrub: create_pool testpool 1 1 2024-03-25T23:24:40.161 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-03-25T23:24:40.616 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'testpool' created 2024-03-25T23:24:40.638 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T23:24:41.640 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:498: TEST_auto_repair_bluestore_scrub: ceph osd pool set testpool size 2 2024-03-25T23:24:42.444 INFO:tasks.workunit.client.0.smithi129.stderr:set pool 1 size to 2 2024-03-25T23:24:42.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:499: TEST_auto_repair_bluestore_scrub: wait_for_clean 2024-03-25T23:24:42.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:24:42.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:24:42.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:24:42.464 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:24:42.465 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:24:42.465 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:24:42.465 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:24:42.466 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:24:42.466 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:24:42.582 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:24:42.583 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:24:42.583 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:24:42.583 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:24:42.583 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:24:42.583 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:24:42.997 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:24:42.998 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:24:42.998 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-25T23:24:42.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:24:42.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:24:42.998 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:24:43.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836505 2024-03-25T23:24:43.141 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836505 2024-03-25T23:24:43.141 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505' 2024-03-25T23:24:43.141 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:24:43.141 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:24:43.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672974 2024-03-25T23:24:43.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672974 2024-03-25T23:24:43.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672974' 2024-03-25T23:24:43.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:24:43.284 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:24:43.430 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509448 2024-03-25T23:24:43.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509448 2024-03-25T23:24:43.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672974 2-64424509448' 2024-03-25T23:24:43.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:24:43.431 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836505 2024-03-25T23:24:43.431 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:24:43.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:24:43.434 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836505 2024-03-25T23:24:43.434 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:24:43.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836505 2024-03-25T23:24:43.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-03-25T23:24:43.435 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 21474836505 2024-03-25T23:24:43.435 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:24:43.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836505 2024-03-25T23:24:43.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:24:44.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:24:44.732 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:24:45.059 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836505 2024-03-25T23:24:45.059 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:24:46.061 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-25T23:24:46.062 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:24:46.377 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836505 -lt 21474836505 2024-03-25T23:24:46.377 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:24:46.378 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672974 2024-03-25T23:24:46.378 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:24:46.378 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:24:46.379 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672974 2024-03-25T23:24:46.380 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:24:46.381 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672974 2024-03-25T23:24:46.381 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-03-25T23:24:46.381 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 42949672974 2024-03-25T23:24:46.381 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:24:46.695 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672975 -lt 42949672974 2024-03-25T23:24:46.696 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:24:46.696 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:24:46.697 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509448 2024-03-25T23:24:46.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:24:46.698 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509448 2024-03-25T23:24:46.699 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:24:46.700 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509448 2024-03-25T23:24:46.700 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 64424509448 2024-03-25T23:24:46.700 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-03-25T23:24:46.700 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:24:47.020 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509448 -lt 64424509448 2024-03-25T23:24:47.021 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:24:47.021 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:24:47.021 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:24:47.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-25T23:24:47.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:24:47.454 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:24:47.454 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:24:47.454 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:24:47.454 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:24:47.454 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:24:47.455 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:24:47.796 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-25T23:24:47.797 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:24:47.797 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:24:47.797 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:24:48.159 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-25T23:24:48.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:24:48.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:24:48.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:502: TEST_auto_repair_bluestore_scrub: local payload=ABCDEF 2024-03-25T23:24:48.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:503: TEST_auto_repair_bluestore_scrub: echo ABCDEF 2024-03-25T23:24:48.161 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:504: TEST_auto_repair_bluestore_scrub: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-25T23:24:48.330 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:508: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2024-03-25T23:24:48.331 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-25T23:24:48.331 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-25T23:24:48.331 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-25T23:24:48.331 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-25T23:24:48.331 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-25T23:24:48.331 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-25T23:24:48.331 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-25T23:24:48.631 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-25T23:24:48.632 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-25T23:24:48.632 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-25T23:24:48.938 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:508: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-03-25T23:24:48.938 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:24:48.939 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:24:48.939 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-25T23:24:48.939 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:24:48.939 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-03-25T23:24:48.939 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:24:48.939 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:24:48.940 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-25T23:24:48.940 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:24:48.940 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-25T23:24:48.940 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:24:48.940 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:24:48.940 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:24:48.940 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:24:48.940 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:24:49.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:24:49.247 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-03-25T23:24:49.247 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:24:49.247 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:24:49.247 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-25T23:24:49.247 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:24:49.247 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-25T23:24:49.247 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-03-25T23:24:50.793 INFO:tasks.workunit.client.0.smithi129.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-03-25T23:24:51.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-25T23:24:51.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:24:51.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:24:51.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-25T23:24:51.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:24:51.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-25T23:24:51.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:24:51.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:24:51.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:24:51.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:24:51.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-25T23:24:51.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-25T23:24:51.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:24:51.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:24:51.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:24:51.347 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:24:51.347 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:24:51.347 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:24:51.347 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:24:51.347 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:24:51.347 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:24:51.348 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:24:51.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:24:51.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:24:51.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:24:51.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:24:51.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:24:51.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:24:51.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:24:51.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:24:51.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:24:51.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-03-25T23:24:51.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-25T23:24:51.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-25T23:24:51.350 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.0 2024-03-25T23:24:51.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-25T23:24:51.350 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-25T23:24:51.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-25T23:24:51.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:24:51.351 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:24:51.351 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:24:51.386 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:24:51.387+0000 7f797c87d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:24:51.388 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:24:51.387+0000 7f797c87d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:24:51.390 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:24:51.391+0000 7f797c87d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:24:51.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-25T23:24:51.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:24:51.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T23:24:51.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:24:51.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:24:51.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:24:51.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:24:51.698 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-25T23:24:51.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:24:51.699 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:24:51.916 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:24:51.915+0000 7f797c87d7c0 -1 Falling back to public interface 2024-03-25T23:24:52.059 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:24:53.061 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:24:53.061 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:24:53.061 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:24:53.061 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:24:53.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:24:53.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:24:53.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:24:54.372 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:24:54.372 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:24:54.372 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:24:54.372 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:24:54.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:24:54.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:24:54.564 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:24:54.563+0000 7f797c87d7c0 -1 osd.0 21 log_to_monitors true 2024-03-25T23:24:54.695 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:24:55.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:24:55.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:24:55.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:24:55.697 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:24:55.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:24:55.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:24:56.075 INFO:tasks.workunit.client.0.smithi129.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/2870937442,v1:127.0.0.1:6803/2870937442] [v2:127.0.0.1:6804/2870937442,v1:127.0.0.1:6805/2870937442] exists,up 9e4ff428-8798-4568-a998-3f2b8861361b 2024-03-25T23:24:56.076 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:24:56.076 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:24:56.076 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:24:56.076 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:24:56.076 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:24:56.076 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:24:56.076 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:24:56.077 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:24:56.077 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:24:56.077 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:24:56.077 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:24:56.077 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:24:56.077 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:24:56.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:24:56.190 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:24:56.190 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:24:56.190 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:24:56.190 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:24:56.190 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:24:56.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:24:56.505 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:24:56.505 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-25T23:24:56.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:24:56.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:24:56.506 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:24:56.683 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-03-25T23:24:56.683 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-03-25T23:24:56.683 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-03-25T23:24:56.683 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:24:56.683 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:24:56.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672978 2024-03-25T23:24:56.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672978 2024-03-25T23:24:56.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672978' 2024-03-25T23:24:56.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:24:56.814 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:24:56.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509452 2024-03-25T23:24:56.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509452 2024-03-25T23:24:56.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672978 2-64424509452' 2024-03-25T23:24:56.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:24:56.952 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-03-25T23:24:56.953 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:24:56.953 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:24:56.954 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-03-25T23:24:56.954 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:24:56.955 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-03-25T23:24:56.955 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-03-25T23:24:56.955 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 107374182402 2024-03-25T23:24:56.956 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:24:57.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-03-25T23:24:57.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:24:57.310 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672978 2024-03-25T23:24:57.310 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:24:57.312 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:24:57.313 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672978 2024-03-25T23:24:57.313 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:24:57.314 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672978 2024-03-25T23:24:57.314 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-03-25T23:24:57.314 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 42949672978 2024-03-25T23:24:57.315 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:24:57.640 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672978 -lt 42949672978 2024-03-25T23:24:57.641 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:24:57.642 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509452 2024-03-25T23:24:57.642 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:24:57.643 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:24:57.644 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509452 2024-03-25T23:24:57.644 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:24:57.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509452 2024-03-25T23:24:57.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-03-25T23:24:57.645 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 64424509452 2024-03-25T23:24:57.646 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:24:57.958 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509452 -lt 64424509452 2024-03-25T23:24:57.959 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:24:57.960 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:24:57.960 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:24:58.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-25T23:24:58.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:24:58.349 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:24:58.349 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:24:58.349 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:24:58.349 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:24:58.350 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:24:58.350 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:24:58.663 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-25T23:24:58.663 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:24:58.664 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:24:58.664 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:24:59.016 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-25T23:24:59.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:24:59.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:24:59.017 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:510: TEST_auto_repair_bluestore_scrub: get_pg testpool SOMETHING 2024-03-25T23:24:59.017 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-03-25T23:24:59.017 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-25T23:24:59.017 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool SOMETHING 2024-03-25T23:24:59.017 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-25T23:24:59.368 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:510: TEST_auto_repair_bluestore_scrub: local pgid=1.0 2024-03-25T23:24:59.369 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:511: TEST_auto_repair_bluestore_scrub: get_primary testpool SOMETHING 2024-03-25T23:24:59.369 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-25T23:24:59.369 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-25T23:24:59.370 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-25T23:24:59.370 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-25T23:24:59.694 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:511: TEST_auto_repair_bluestore_scrub: local primary=1 2024-03-25T23:24:59.694 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:512: TEST_auto_repair_bluestore_scrub: get_last_scrub_stamp 1.0 2024-03-25T23:24:59.695 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-25T23:24:59.695 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:24:59.695 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-25T23:24:59.695 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:25:00.002 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:512: TEST_auto_repair_bluestore_scrub: local last_scrub_stamp=2024-03-25T23:24:40.519761+0000 2024-03-25T23:25:00.002 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:513: TEST_auto_repair_bluestore_scrub: ceph tell 1.0 schedule-scrub 2024-03-25T23:25:00.134 INFO:tasks.workunit.client.0.smithi129.stdout:{ 2024-03-25T23:25:00.134 INFO:tasks.workunit.client.0.smithi129.stdout: "deep": false, 2024-03-25T23:25:00.134 INFO:tasks.workunit.client.0.smithi129.stdout: "must": false, 2024-03-25T23:25:00.134 INFO:tasks.workunit.client.0.smithi129.stdout: "stamp": "2024-03-18T23:23:20.138709+0000" 2024-03-25T23:25:00.134 INFO:tasks.workunit.client.0.smithi129.stdout:} 2024-03-25T23:25:00.148 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:516: TEST_auto_repair_bluestore_scrub: wait_for_scrub 1.0 2024-03-25T23:24:40.519761+0000 2024-03-25T23:25:00.148 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-03-25T23:25:00.148 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-25T23:24:40.519761+0000 2024-03-25T23:25:00.148 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-25T23:25:00.148 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-25T23:25:00.148 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:25:00.149 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-25T23:25:00.149 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-25T23:25:00.149 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:25:00.149 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:25:00.149 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-25T23:25:00.429 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:24:40.519761+0000 '>' 2024-03-25T23:24:40.519761+0000 2024-03-25T23:25:00.430 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:25:01.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:25:01.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:25:01.432 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-25T23:25:01.432 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-25T23:25:01.432 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:25:01.433 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:25:01.433 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-25T23:25:01.736 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:24:40.519761+0000 '>' 2024-03-25T23:24:40.519761+0000 2024-03-25T23:25:01.736 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:25:02.737 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:25:02.738 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:25:02.738 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-25T23:25:02.738 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-25T23:25:02.738 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:25:02.739 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:25:02.739 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-25T23:25:03.056 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:24:40.519761+0000 '>' 2024-03-25T23:24:40.519761+0000 2024-03-25T23:25:03.056 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:25:04.058 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:25:04.058 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:25:04.058 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-25T23:25:04.058 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-25T23:25:04.058 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:25:04.059 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:25:04.059 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-25T23:25:04.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:24:40.519761+0000 '>' 2024-03-25T23:24:40.519761+0000 2024-03-25T23:25:04.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:25:05.380 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:25:05.380 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:25:05.381 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-25T23:25:05.381 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-25T23:25:05.381 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:25:05.382 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-25T23:25:05.382 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:25:05.692 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:25:01.775096+0000 '>' 2024-03-25T23:24:40.519761+0000 2024-03-25T23:25:05.692 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-25T23:25:05.692 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:517: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-03-25T23:25:05.975 INFO:tasks.workunit.client.0.smithi129.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-25T23:25:05.975 INFO:tasks.workunit.client.0.smithi129.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-03-25T23:25:01.986155+0000 21'1 26:64 [1,0] 1 [1,0] 1 21'1 2024-03-25T23:25:01.775096+0000 21'1 2024-03-25T23:25:01.775096+0000 0 1 periodic scrub scheduled @ 2024-03-26T23:25:01.775096+0000 1 0 2024-03-25T23:25:05.975 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:25:05.975 INFO:tasks.workunit.client.0.smithi129.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-25T23:25:05.975 INFO:tasks.workunit.client.0.smithi129.stderr:dumped pgs 2024-03-25T23:25:05.990 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:519: TEST_auto_repair_bluestore_scrub: sleep 2 2024-03-25T23:25:07.992 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:520: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-03-25T23:25:08.288 INFO:tasks.workunit.client.0.smithi129.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-25T23:25:08.288 INFO:tasks.workunit.client.0.smithi129.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-03-25T23:25:01.986155+0000 21'1 26:64 [1,0] 1 [1,0] 1 21'1 2024-03-25T23:25:01.775096+0000 21'1 2024-03-25T23:25:01.775096+0000 0 1 periodic scrub scheduled @ 2024-03-26T23:25:01.775096+0000 1 0 2024-03-25T23:25:08.288 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:25:08.288 INFO:tasks.workunit.client.0.smithi129.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-25T23:25:08.289 INFO:tasks.workunit.client.0.smithi129.stderr:dumped pgs 2024-03-25T23:25:08.305 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:521: TEST_auto_repair_bluestore_scrub: sleep 2 2024-03-25T23:25:10.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:522: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-03-25T23:25:10.606 INFO:tasks.workunit.client.0.smithi129.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-25T23:25:10.607 INFO:tasks.workunit.client.0.smithi129.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-03-25T23:25:01.986155+0000 21'1 26:64 [1,0] 1 [1,0] 1 21'1 2024-03-25T23:25:01.775096+0000 21'1 2024-03-25T23:25:01.775096+0000 0 1 periodic scrub scheduled @ 2024-03-26T23:25:01.775096+0000 1 0 2024-03-25T23:25:10.607 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:25:10.607 INFO:tasks.workunit.client.0.smithi129.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-25T23:25:10.607 INFO:tasks.workunit.client.0.smithi129.stderr:dumped pgs 2024-03-25T23:25:10.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:523: TEST_auto_repair_bluestore_scrub: sleep 5 2024-03-25T23:25:15.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:524: TEST_auto_repair_bluestore_scrub: wait_for_clean 2024-03-25T23:25:15.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:25:15.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:25:15.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:25:15.626 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:25:15.627 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:25:15.627 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:25:15.628 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:25:15.628 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:25:15.628 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:25:15.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:25:15.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:25:15.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:25:15.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:25:15.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:25:15.757 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:25:16.066 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:25:16.066 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:25:16.066 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-25T23:25:16.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:25:16.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:25:16.067 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:25:16.226 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182407 2024-03-25T23:25:16.226 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182407 2024-03-25T23:25:16.226 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182407' 2024-03-25T23:25:16.226 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:25:16.226 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:25:16.366 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672983 2024-03-25T23:25:16.366 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672983 2024-03-25T23:25:16.366 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182407 1-42949672983' 2024-03-25T23:25:16.366 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:25:16.366 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:25:16.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509457 2024-03-25T23:25:16.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509457 2024-03-25T23:25:16.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182407 1-42949672983 2-64424509457' 2024-03-25T23:25:16.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:25:16.509 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182407 2024-03-25T23:25:16.509 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:25:16.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:25:16.510 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182407 2024-03-25T23:25:16.510 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:25:16.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182407 2024-03-25T23:25:16.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182407' 2024-03-25T23:25:16.512 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 107374182407 2024-03-25T23:25:16.512 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:25:16.807 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182406 -lt 107374182407 2024-03-25T23:25:16.807 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:25:17.808 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:25:17.809 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:25:18.127 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182407 -lt 107374182407 2024-03-25T23:25:18.127 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:25:18.129 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672983 2024-03-25T23:25:18.129 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:25:18.130 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:25:18.131 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672983 2024-03-25T23:25:18.131 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:25:18.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672983 2024-03-25T23:25:18.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-03-25T23:25:18.132 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 42949672983 2024-03-25T23:25:18.133 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:25:18.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672983 -lt 42949672983 2024-03-25T23:25:18.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:25:18.497 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509457 2024-03-25T23:25:18.497 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:25:18.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:25:18.500 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509457 2024-03-25T23:25:18.500 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:25:18.501 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509457 2024-03-25T23:25:18.501 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-03-25T23:25:18.501 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 64424509457 2024-03-25T23:25:18.501 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:25:18.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509457 -lt 64424509457 2024-03-25T23:25:18.833 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:25:18.834 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:25:18.834 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:25:19.233 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-25T23:25:19.234 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:25:19.234 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:25:19.235 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:25:19.235 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:25:19.235 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:25:19.235 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:25:19.235 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:25:19.570 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-25T23:25:19.571 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:25:19.571 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:25:19.571 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:25:19.934 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-25T23:25:19.934 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:25:19.934 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:25:19.934 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:525: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-03-25T23:25:20.252 INFO:tasks.workunit.client.0.smithi129.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-25T23:25:20.252 INFO:tasks.workunit.client.0.smithi129.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-03-25T23:25:01.986155+0000 21'1 26:64 [1,0] 1 [1,0] 1 21'1 2024-03-25T23:25:01.775096+0000 21'1 2024-03-25T23:25:01.775096+0000 0 1 periodic scrub scheduled @ 2024-03-26T23:25:01.775096+0000 1 0 2024-03-25T23:25:20.252 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:25:20.252 INFO:tasks.workunit.client.0.smithi129.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-25T23:25:20.253 INFO:tasks.workunit.client.0.smithi129.stderr:dumped pgs 2024-03-25T23:25:20.270 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:528: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2024-03-25T23:25:20.270 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-25T23:25:20.270 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-25T23:25:20.271 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-25T23:25:20.271 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-25T23:25:20.271 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-25T23:25:20.271 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-25T23:25:20.271 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-25T23:25:20.572 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-25T23:25:20.573 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-25T23:25:20.573 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-25T23:25:20.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:528: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-25T23:25:20.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:25:20.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:25:20.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-25T23:25:20.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:25:20.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-25T23:25:20.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:25:20.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:25:20.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-25T23:25:20.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:25:20.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-25T23:25:20.898 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:25:20.899 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:25:20.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:25:20.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:25:20.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:25:21.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:25:21.208 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-25T23:25:21.208 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:25:21.208 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:25:21.208 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-25T23:25:21.208 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:25:21.208 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-25T23:25:21.208 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-03-25T23:25:21.389 INFO:tasks.workunit.client.0.smithi129.stdout:_ 2024-03-25T23:25:21.389 INFO:tasks.workunit.client.0.smithi129.stdout:snapset 2024-03-25T23:25:21.714 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-25T23:25:21.714 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:25:21.714 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:25:21.714 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-25T23:25:21.714 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:25:21.714 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-25T23:25:21.714 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:25:21.714 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:25:21.714 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:25:21.714 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:25:21.714 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-25T23:25:21.715 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-25T23:25:21.715 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:25:21.715 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:25:21.715 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:25:21.715 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:25:21.715 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:25:21.715 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:25:21.715 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:25:21.715 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:25:21.715 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:25:21.715 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:25:21.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:25:21.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:25:21.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:25:21.717 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:25:21.717 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:25:21.717 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:25:21.717 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:25:21.717 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:25:21.717 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:25:21.717 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-03-25T23:25:21.717 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-25T23:25:21.717 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-25T23:25:21.717 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.0 2024-03-25T23:25:21.718 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-25T23:25:21.718 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-25T23:25:21.718 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-25T23:25:21.719 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:25:21.719 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:25:21.719 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:25:21.755 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:25:21.754+0000 7f435db1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:25:21.759 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:25:21.758+0000 7f435db1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:25:21.761 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:25:21.758+0000 7f435db1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:25:22.039 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-25T23:25:22.039 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:25:22.039 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T23:25:22.039 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:25:22.039 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:25:22.039 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:25:22.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:25:22.040 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-25T23:25:22.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:25:22.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:25:22.299 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:25:22.298+0000 7f435db1d7c0 -1 Falling back to public interface 2024-03-25T23:25:22.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:25:23.384 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:25:23.384 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:25:23.384 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:25:23.384 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:25:23.385 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:25:23.385 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:25:23.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:25:24.715 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:25:24.715 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:25:24.715 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:25:24.715 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:25:24.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:25:24.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:25:25.039 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:25:25.291 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:25:25.290+0000 7f435db1d7c0 -1 osd.0 26 log_to_monitors true 2024-03-25T23:25:26.041 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:25:26.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:25:26.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:25:26.042 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:25:26.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:25:26.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:25:26.405 INFO:tasks.workunit.client.0.smithi129.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/556210953,v1:127.0.0.1:6803/556210953] [v2:127.0.0.1:6804/556210953,v1:127.0.0.1:6805/556210953] exists,up 9e4ff428-8798-4568-a998-3f2b8861361b 2024-03-25T23:25:26.405 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:25:26.405 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:25:26.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:25:26.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:25:26.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:25:26.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:25:26.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:25:26.406 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:25:26.406 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:25:26.406 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:25:26.407 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:25:26.407 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:25:26.407 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:25:26.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:25:26.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:25:26.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:25:26.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:25:26.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:25:26.515 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:25:26.822 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:25:26.822 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:25:26.822 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-25T23:25:26.822 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:25:26.822 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:25:26.822 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:25:26.992 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-03-25T23:25:26.992 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-03-25T23:25:26.992 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882' 2024-03-25T23:25:26.992 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:25:26.992 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:25:27.133 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672986 2024-03-25T23:25:27.133 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672986 2024-03-25T23:25:27.134 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672986' 2024-03-25T23:25:27.134 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:25:27.134 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:25:27.277 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509460 2024-03-25T23:25:27.277 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509460 2024-03-25T23:25:27.277 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672986 2-64424509460' 2024-03-25T23:25:27.277 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:25:27.278 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018882 2024-03-25T23:25:27.278 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:25:27.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:25:27.280 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018882 2024-03-25T23:25:27.280 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:25:27.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-03-25T23:25:27.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-03-25T23:25:27.281 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 128849018882 2024-03-25T23:25:27.281 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:25:27.573 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-03-25T23:25:27.573 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:25:27.574 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672986 2024-03-25T23:25:27.574 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:25:27.575 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:25:27.576 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672986 2024-03-25T23:25:27.576 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:25:27.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672986 2024-03-25T23:25:27.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2024-03-25T23:25:27.577 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 42949672986 2024-03-25T23:25:27.577 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:25:27.893 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672985 -lt 42949672986 2024-03-25T23:25:27.894 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:25:28.895 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:25:28.895 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:25:29.215 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672987 -lt 42949672986 2024-03-25T23:25:29.215 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:25:29.216 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:25:29.216 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509460 2024-03-25T23:25:29.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:25:29.218 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:25:29.222 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509460 2024-03-25T23:25:29.222 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509460 2024-03-25T23:25:29.222 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-03-25T23:25:29.222 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 64424509460 2024-03-25T23:25:29.222 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:25:29.534 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509460 -lt 64424509460 2024-03-25T23:25:29.534 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:25:29.535 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:25:29.535 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:25:29.925 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-25T23:25:29.925 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:25:29.926 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:25:29.926 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:25:29.926 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:25:29.926 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:25:29.926 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:25:29.926 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:25:30.252 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-25T23:25:30.253 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:25:30.253 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:25:30.254 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:25:30.635 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-25T23:25:30.635 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:25:30.635 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:25:30.636 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:529: TEST_auto_repair_bluestore_scrub: rados --pool testpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-25T23:25:30.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:530: TEST_auto_repair_bluestore_scrub: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-25T23:25:30.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:531: TEST_auto_repair_bluestore_scrub: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-03-25T23:25:30.705 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:25:00.739+0000 7f53e8fa8640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: DEEP_SCRUB_ON_ERROR ] planned DEEP_SCRUB_ON_ERROR] scrubber: scrub_finish before flags: DEEP_SCRUB_ON_ERROR. repair state: no-repair. deep_scrub_on_error: 1 2024-03-25T23:25:30.705 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:25:00.739+0000 7f53e8fa8640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: DEEP_SCRUB_ON_ERROR ] ] scrubber: scrub_finish Try to auto repair after scrub errors 2024-03-25T23:25:30.705 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:25:00.739+0000 7f53e8fa8640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-03-25T23:25:30.705 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:25:01.771+0000 7f53e8fa8640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-03-25T23:25:30.706 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:25:01.771+0000 7f53e8fa8640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-03-25T23:25:30.706 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:25:01.771+0000 7f53e8fa8640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-03-25T23:25:30.706 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:25:01.771+0000 7f53e8fa8640 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-03-25T23:25:30.706 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:25:01.771+0000 7f53e8fa8640 19 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-25T23:25:30.706 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: ceph pg 1.0 query 2024-03-25T23:25:30.706 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: jq .info.stats.stat_sum.num_objects_repaired 2024-03-25T23:25:30.847 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: COUNT=1 2024-03-25T23:25:30.848 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:535: TEST_auto_repair_bluestore_scrub: test 1 = 1 2024-03-25T23:25:30.848 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-25T23:25:30.848 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-25T23:25:30.848 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T23:25:30.848 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-25T23:25:30.848 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:25:30.848 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:25:30.848 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:25:30.848 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:25:30.848 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:25:31.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:25:31.158 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T23:25:31.159 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T23:25:31.159 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T23:25:31.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T23:25:31.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T23:25:31.161 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T23:25:31.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:25:31.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T23:25:31.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T23:25:31.163 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:25:31.164 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T23:25:31.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T23:25:31.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-25T23:25:31.186 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T23:25:31.186 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:25:31.186 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:25:31.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-25T23:25:31.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T23:25:31.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T23:25:31.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-25T23:25:31.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-25T23:25:31.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-25T23:25:31.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-25T23:25:31.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-25T23:25:31.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T23:25:31.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-25T23:25:31.189 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:25:31.189 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:25:31.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:25:31.190 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:25:31.190 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:25:31.191 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:25:31.192 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T23:25:31.193 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T23:25:31.193 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T23:25:31.195 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T23:25:31.195 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T23:25:31.195 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T23:25:31.196 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:25:31.196 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T23:25:31.196 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T23:25:31.197 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:25:31.198 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T23:25:31.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T23:25:31.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-25T23:25:31.201 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T23:25:31.201 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:25:31.201 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:25:31.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-25T23:25:31.202 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T23:25:31.202 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T23:25:31.202 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-25T23:25:31.203 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-25T23:25:31.204 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:25:31.204 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:25:31.204 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61636 2024-03-25T23:25:31.205 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-25T23:25:31.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-25T23:25:31.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-25T23:25:31.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-25T23:25:31.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_tag td/osd-scrub-repair 2024-03-25T23:25:31.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:383: TEST_auto_repair_bluestore_tag: local dir=td/osd-scrub-repair 2024-03-25T23:25:31.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:384: TEST_auto_repair_bluestore_tag: local poolname=testpool 2024-03-25T23:25:31.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:387: TEST_auto_repair_bluestore_tag: run_mon td/osd-scrub-repair a 2024-03-25T23:25:31.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-25T23:25:31.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-25T23:25:31.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-25T23:25:31.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-25T23:25:31.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-25T23:25:31.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-25T23:25:31.689 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-25T23:25:31.689 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:25:31.689 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:25:31.690 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:25:31.690 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:25:31.690 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:25:31.690 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:25:31.690 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-25T23:25:32.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-25T23:25:32.018 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-25T23:25:32.018 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T23:25:32.018 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T23:25:32.018 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-25T23:25:32.019 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-25T23:25:32.019 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T23:25:32.019 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T23:25:32.019 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T23:25:32.019 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T23:25:32.019 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:25:32.019 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:25:32.020 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-25T23:25:32.020 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T23:25:32.020 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get fsid 2024-03-25T23:25:32.135 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-25T23:25:32.135 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T23:25:32.135 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T23:25:32.135 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-25T23:25:32.136 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-25T23:25:32.136 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T23:25:32.136 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T23:25:32.136 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T23:25:32.137 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T23:25:32.137 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:25:32.137 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:25:32.137 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-25T23:25:32.137 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T23:25:32.137 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get mon_host 2024-03-25T23:25:32.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:388: TEST_auto_repair_bluestore_tag: run_mgr td/osd-scrub-repair x 2024-03-25T23:25:32.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-25T23:25:32.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-25T23:25:32.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-25T23:25:32.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-25T23:25:32.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-25T23:25:32.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-25T23:25:33.119 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-25T23:25:33.119 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:25:33.119 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:25:33.119 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:25:33.119 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:25:33.119 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:25:33.119 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:25:33.120 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T23:25:33.122 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T23:25:33.156 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:395: TEST_auto_repair_bluestore_tag: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-03-25T23:25:33.156 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: seq 0 2 2024-03-25T23:25:33.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-03-25T23:25:33.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-25T23:25:33.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:25:33.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:25:33.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-25T23:25:33.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:25:33.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-25T23:25:33.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:25:33.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:25:33.161 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:25:33.161 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:25:33.161 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-25T23:25:33.161 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-25T23:25:33.161 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:25:33.161 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:25:33.161 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:25:33.161 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:25:33.161 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:25:33.161 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:25:33.161 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:25:33.161 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:25:33.161 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:25:33.161 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:25:33.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:25:33.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:25:33.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:25:33.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:25:33.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:25:33.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:25:33.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:25:33.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:25:33.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:25:33.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:25:33.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:25:33.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-03-25T23:25:33.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-25T23:25:33.163 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:25:33.163 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7fcefe91-10b2-4f70-ac2c-7f0c3783e583 2024-03-25T23:25:33.163 INFO:tasks.workunit.client.0.smithi129.stdout:add osd0 7fcefe91-10b2-4f70-ac2c-7f0c3783e583 2024-03-25T23:25:33.164 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 7fcefe91-10b2-4f70-ac2c-7f0c3783e583' 2024-03-25T23:25:33.164 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:25:33.181 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDtBwJmKdvsChAA16wMTTqEwIY4YyHOrM8qVQ== 2024-03-25T23:25:33.181 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDtBwJmKdvsChAA16wMTTqEwIY4YyHOrM8qVQ=="}' 2024-03-25T23:25:33.181 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7fcefe91-10b2-4f70-ac2c-7f0c3783e583 -i td/osd-scrub-repair/0/new.json 2024-03-25T23:25:33.444 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:25:33.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-25T23:25:33.465 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQDtBwJmKdvsChAA16wMTTqEwIY4YyHOrM8qVQ== --osd-uuid 7fcefe91-10b2-4f70-ac2c-7f0c3783e583 2024-03-25T23:25:33.501 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:25:33.498+0000 7f7de7d347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:25:33.511 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:25:33.510+0000 7f7de7d347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:25:33.513 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:25:33.510+0000 7f7de7d347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:25:33.513 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:25:33.510+0000 7f7de7d347c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-25T23:25:33.513 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:25:33.510+0000 7f7de7d347c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-25T23:25:33.513 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:25:33.510+0000 7f7de7d347c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-25T23:25:33.612 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:25:33.610+0000 7f7de7d347c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-25T23:25:39.940 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-25T23:25:39.940 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:25:39.942 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd0 key to auth repository 2024-03-25T23:25:39.942 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-25T23:25:39.943 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:25:40.314 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.0 2024-03-25T23:25:40.315 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-25T23:25:40.315 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:25:40.316 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:25:40.316 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-25T23:25:40.316 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:25:40.350 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:25:40.350+0000 7f19f97757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:25:40.377 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:25:40.374+0000 7f19f97757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:25:40.379 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:25:40.378+0000 7f19f97757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:25:40.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-25T23:25:40.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:25:40.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T23:25:40.651 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:25:40.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:25:40.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:25:40.652 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:25:40.652 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:25:40.652 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:25:40.652 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:25:40.936 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:25:40.934+0000 7f19f97757c0 -1 Falling back to public interface 2024-03-25T23:25:40.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:25:41.990 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:25:41.990 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:25:41.990 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:25:41.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:25:41.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:25:41.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:25:42.313 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:25:43.312 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:25:43.312 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:25:43.313 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:25:43.313 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:25:43.313 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:25:43.313 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:25:43.512 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:25:43.514+0000 7f19f97757c0 -1 osd.0 0 log_to_monitors true 2024-03-25T23:25:43.632 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:25:44.633 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:25:44.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:25:44.634 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:25:44.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:25:44.635 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:25:44.635 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:25:44.959 INFO:tasks.workunit.client.0.smithi129.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3449236396,v1:127.0.0.1:6803/3449236396] [v2:127.0.0.1:6804/3449236396,v1:127.0.0.1:6805/3449236396] exists,up 7fcefe91-10b2-4f70-ac2c-7f0c3783e583 2024-03-25T23:25:44.961 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:25:44.961 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:25:44.961 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:25:44.961 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-03-25T23:25:44.961 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-25T23:25:44.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:25:44.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:25:44.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-25T23:25:44.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:25:44.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:25:44.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:25:44.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:25:44.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:25:44.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:25:44.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-25T23:25:44.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-25T23:25:44.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:25:44.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:25:44.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:25:44.963 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:25:44.963 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:25:44.963 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:25:44.965 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:25:44.965 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:25:44.965 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:25:44.965 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:25:44.965 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:25:44.965 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:25:44.965 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:25:44.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:25:44.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:25:44.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:25:44.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:25:44.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:25:44.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:25:44.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:25:44.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:25:44.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-03-25T23:25:44.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-25T23:25:44.966 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:25:44.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d151a968-2143-4421-96db-bbec36a126bc 2024-03-25T23:25:44.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 d151a968-2143-4421-96db-bbec36a126bc' 2024-03-25T23:25:44.967 INFO:tasks.workunit.client.0.smithi129.stdout:add osd1 d151a968-2143-4421-96db-bbec36a126bc 2024-03-25T23:25:44.967 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:25:44.983 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD4BwJmwJ7KOhAAtNp1+0WQXjm5vK6THzVgUw== 2024-03-25T23:25:44.983 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD4BwJmwJ7KOhAAtNp1+0WQXjm5vK6THzVgUw=="}' 2024-03-25T23:25:44.983 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d151a968-2143-4421-96db-bbec36a126bc -i td/osd-scrub-repair/1/new.json 2024-03-25T23:25:45.322 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:25:45.338 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-25T23:25:45.339 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQD4BwJmwJ7KOhAAtNp1+0WQXjm5vK6THzVgUw== --osd-uuid d151a968-2143-4421-96db-bbec36a126bc 2024-03-25T23:25:45.376 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:25:45.378+0000 7fb59eff67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:25:45.415 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:25:45.418+0000 7fb59eff67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:25:45.416 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:25:45.418+0000 7fb59eff67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:25:45.417 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:25:45.418+0000 7fb59eff67c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-25T23:25:45.417 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:25:45.418+0000 7fb59eff67c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-25T23:25:45.417 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:25:45.418+0000 7fb59eff67c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-25T23:25:45.496 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:25:45.498+0000 7fb59eff67c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-25T23:25:52.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-25T23:25:52.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:25:52.133 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd1 key to auth repository 2024-03-25T23:25:52.133 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-25T23:25:52.133 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:25:52.546 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.1 2024-03-25T23:25:52.547 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-25T23:25:52.547 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-25T23:25:52.547 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:25:52.547 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:25:52.548 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:25:52.584 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:25:52.585+0000 7f6aa51887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:25:52.613 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:25:52.613+0000 7f6aa51887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:25:52.615 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:25:52.617+0000 7f6aa51887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:25:52.880 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-25T23:25:52.880 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:25:52.880 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:25:52.881 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T23:25:52.881 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:25:52.881 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:25:52.881 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:25:52.881 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:25:52.881 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:25:52.882 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:25:53.163 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:25:53.165+0000 7f6aa51887c0 -1 Falling back to public interface 2024-03-25T23:25:53.212 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:25:54.213 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:25:54.214 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:25:54.214 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:25:54.214 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:25:54.214 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:25:54.215 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:25:54.525 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:25:55.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:25:55.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:25:55.528 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:25:55.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:25:55.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:25:55.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:25:55.733 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:25:55.733+0000 7f6aa51887c0 -1 osd.1 0 log_to_monitors true 2024-03-25T23:25:55.859 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:25:56.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:25:56.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:25:56.861 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:25:56.862 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:25:56.862 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:25:56.862 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:25:57.181 INFO:tasks.workunit.client.0.smithi129.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/765090078,v1:127.0.0.1:6811/765090078] [v2:127.0.0.1:6812/765090078,v1:127.0.0.1:6813/765090078] exists,up d151a968-2143-4421-96db-bbec36a126bc 2024-03-25T23:25:57.185 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:25:57.185 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:25:57.185 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:25:57.185 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-03-25T23:25:57.185 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-25T23:25:57.185 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:25:57.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:25:57.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-25T23:25:57.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:25:57.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-25T23:25:57.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:25:57.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:25:57.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:25:57.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:25:57.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-25T23:25:57.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-25T23:25:57.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:25:57.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:25:57.187 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:25:57.187 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:25:57.187 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:25:57.187 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:25:57.187 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:25:57.187 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:25:57.187 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:25:57.187 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:25:57.187 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:25:57.187 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:25:57.187 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:25:57.187 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:25:57.187 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:25:57.187 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:25:57.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:25:57.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:25:57.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:25:57.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:25:57.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:25:57.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-03-25T23:25:57.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-25T23:25:57.188 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:25:57.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8e594b1c-1705-41d6-8026-4ccf34eabb24 2024-03-25T23:25:57.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 8e594b1c-1705-41d6-8026-4ccf34eabb24' 2024-03-25T23:25:57.189 INFO:tasks.workunit.client.0.smithi129.stdout:add osd2 8e594b1c-1705-41d6-8026-4ccf34eabb24 2024-03-25T23:25:57.189 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:25:57.208 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAFCAJmltGaDBAASK5N4XtXFxFoWmooRA7kMw== 2024-03-25T23:25:57.208 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAFCAJmltGaDBAASK5N4XtXFxFoWmooRA7kMw=="}' 2024-03-25T23:25:57.208 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8e594b1c-1705-41d6-8026-4ccf34eabb24 -i td/osd-scrub-repair/2/new.json 2024-03-25T23:25:57.574 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:25:57.591 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-25T23:25:57.592 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQAFCAJmltGaDBAASK5N4XtXFxFoWmooRA7kMw== --osd-uuid 8e594b1c-1705-41d6-8026-4ccf34eabb24 2024-03-25T23:25:57.624 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:25:57.625+0000 7f0048f217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:25:57.666 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:25:57.669+0000 7f0048f217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:25:57.668 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:25:57.669+0000 7f0048f217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:25:57.669 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:25:57.669+0000 7f0048f217c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-25T23:25:57.669 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:25:57.669+0000 7f0048f217c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-25T23:25:57.669 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:25:57.669+0000 7f0048f217c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-25T23:25:57.742 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:25:57.741+0000 7f0048f217c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-25T23:26:04.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-25T23:26:04.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:26:04.713 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd2 key to auth repository 2024-03-25T23:26:04.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-25T23:26:04.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:26:05.105 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.2 2024-03-25T23:26:05.106 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-25T23:26:05.106 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-25T23:26:05.106 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:26:05.106 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:26:05.106 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:26:05.139 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:26:05.141+0000 7f23962257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:26:05.174 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:26:05.177+0000 7f23962257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:26:05.175 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:26:05.177+0000 7f23962257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:26:05.449 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-25T23:26:05.449 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:26:05.449 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-25T23:26:05.449 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:26:05.449 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:26:05.449 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:26:05.449 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:26:05.450 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:26:05.450 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:26:05.450 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:26:05.740 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:26:05.741+0000 7f23962257c0 -1 Falling back to public interface 2024-03-25T23:26:05.785 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:26:06.786 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:26:06.786 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:26:06.787 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:26:06.787 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:26:06.787 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:26:06.787 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:26:07.087 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:26:08.089 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:26:08.089 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:26:08.089 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:26:08.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:26:08.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:26:08.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:26:08.473 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:26:08.500 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:26:08.501+0000 7f23962257c0 -1 osd.2 0 log_to_monitors true 2024-03-25T23:26:09.474 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:26:09.474 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:26:09.475 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:26:09.475 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:26:09.475 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:26:09.475 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:26:09.851 INFO:tasks.workunit.client.0.smithi129.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/452965660,v1:127.0.0.1:6819/452965660] [v2:127.0.0.1:6820/452965660,v1:127.0.0.1:6821/452965660] exists,up 8e594b1c-1705-41d6-8026-4ccf34eabb24 2024-03-25T23:26:09.852 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:26:09.852 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:26:09.852 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:26:09.852 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:400: TEST_auto_repair_bluestore_tag: create_pool testpool 1 1 2024-03-25T23:26:09.852 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-03-25T23:26:10.387 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'testpool' created 2024-03-25T23:26:10.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T23:26:11.411 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: TEST_auto_repair_bluestore_tag: ceph osd pool set testpool size 2 2024-03-25T23:26:11.869 INFO:tasks.workunit.client.0.smithi129.stderr:set pool 1 size to 2 2024-03-25T23:26:11.886 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:402: TEST_auto_repair_bluestore_tag: wait_for_clean 2024-03-25T23:26:11.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:26:11.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:26:11.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:26:11.887 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:26:11.888 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:26:11.888 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:26:11.888 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:26:11.888 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:26:11.888 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:26:11.997 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:26:11.997 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:26:11.997 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:26:11.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:26:11.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:26:11.998 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:26:12.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:26:12.391 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:26:12.391 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-25T23:26:12.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:26:12.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:26:12.392 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:26:12.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836487 2024-03-25T23:26:12.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836487 2024-03-25T23:26:12.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836487' 2024-03-25T23:26:12.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:26:12.567 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:26:12.709 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672965 2024-03-25T23:26:12.709 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672965 2024-03-25T23:26:12.709 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836487 1-42949672965' 2024-03-25T23:26:12.709 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:26:12.709 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:26:12.931 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509442 2024-03-25T23:26:12.931 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509442 2024-03-25T23:26:12.932 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836487 1-42949672965 2-64424509442' 2024-03-25T23:26:12.932 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:26:12.932 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836487 2024-03-25T23:26:12.932 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:26:12.934 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:26:12.935 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836487 2024-03-25T23:26:12.935 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:26:12.936 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836487 2024-03-25T23:26:12.936 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-03-25T23:26:12.936 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 21474836487 2024-03-25T23:26:12.936 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:26:13.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836486 -lt 21474836487 2024-03-25T23:26:13.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:26:14.241 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:26:14.241 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:26:14.563 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836487 -lt 21474836487 2024-03-25T23:26:14.563 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:26:14.564 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672965 2024-03-25T23:26:14.564 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:26:14.566 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:26:14.566 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672965 2024-03-25T23:26:14.567 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:26:14.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672965 2024-03-25T23:26:14.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-03-25T23:26:14.569 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 42949672965 2024-03-25T23:26:14.569 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:26:14.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672965 -lt 42949672965 2024-03-25T23:26:14.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:26:14.898 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509442 2024-03-25T23:26:14.899 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:26:14.901 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:26:14.902 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509442 2024-03-25T23:26:14.902 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:26:14.903 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509442 2024-03-25T23:26:14.903 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2024-03-25T23:26:14.903 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 64424509442 2024-03-25T23:26:14.904 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:26:15.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509442 -lt 64424509442 2024-03-25T23:26:15.239 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:26:15.240 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:26:15.240 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:26:15.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-25T23:26:15.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:26:15.615 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:26:15.616 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:26:15.616 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:26:15.616 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:26:15.616 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:26:15.616 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:26:15.959 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-25T23:26:15.959 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:26:15.959 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:26:15.960 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:26:16.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-25T23:26:16.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:26:16.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:26:16.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:405: TEST_auto_repair_bluestore_tag: local payload=ABCDEF 2024-03-25T23:26:16.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: TEST_auto_repair_bluestore_tag: echo ABCDEF 2024-03-25T23:26:16.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:407: TEST_auto_repair_bluestore_tag: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-25T23:26:16.492 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:411: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-03-25T23:26:16.493 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-25T23:26:16.493 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-25T23:26:16.493 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-25T23:26:16.493 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-25T23:26:16.493 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-25T23:26:16.494 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-25T23:26:16.494 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-25T23:26:16.793 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-25T23:26:16.793 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-25T23:26:16.793 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-25T23:26:17.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:411: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-03-25T23:26:17.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:26:17.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:26:17.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-25T23:26:17.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:26:17.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-03-25T23:26:17.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:26:17.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:26:17.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-25T23:26:17.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:26:17.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-25T23:26:17.140 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:26:17.141 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:26:17.141 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:26:17.141 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:26:17.141 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:26:17.450 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:26:17.450 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-03-25T23:26:17.450 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:26:17.450 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:26:17.450 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-25T23:26:17.450 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:26:17.450 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-25T23:26:17.450 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-03-25T23:26:19.057 INFO:tasks.workunit.client.0.smithi129.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-03-25T23:26:19.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-25T23:26:19.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:26:19.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:26:19.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-25T23:26:19.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:26:19.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-25T23:26:19.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:26:19.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:26:19.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:26:19.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:26:19.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-25T23:26:19.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-25T23:26:19.597 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:26:19.597 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:26:19.597 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:26:19.597 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:26:19.597 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:26:19.597 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:26:19.597 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:26:19.597 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:26:19.597 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:26:19.598 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:26:19.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:26:19.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:26:19.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:26:19.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:26:19.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:26:19.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:26:19.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:26:19.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:26:19.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:26:19.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-03-25T23:26:19.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-25T23:26:19.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-25T23:26:19.599 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.0 2024-03-25T23:26:19.600 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-25T23:26:19.600 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-25T23:26:19.600 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-25T23:26:19.601 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:26:19.601 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:26:19.601 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:26:19.636 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:26:19.637+0000 7f46c61287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:26:19.639 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:26:19.641+0000 7f46c61287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:26:19.640 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:26:19.641+0000 7f46c61287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:26:19.918 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-25T23:26:19.918 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:26:19.918 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T23:26:19.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:26:19.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:26:19.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:26:19.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:26:19.919 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-25T23:26:19.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:26:19.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:26:20.180 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:26:20.180+0000 7f46c61287c0 -1 Falling back to public interface 2024-03-25T23:26:20.266 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:26:21.268 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:26:21.268 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:26:21.268 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:26:21.268 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:26:21.268 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:26:21.269 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:26:21.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:26:22.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:26:22.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:26:22.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:26:22.587 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:26:22.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:26:22.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:26:22.847 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:26:22.848+0000 7f46c61287c0 -1 osd.0 21 log_to_monitors true 2024-03-25T23:26:22.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:26:23.913 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:26:23.913 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:26:23.913 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:26:23.913 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:26:23.913 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:26:23.913 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:26:24.235 INFO:tasks.workunit.client.0.smithi129.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/4040155358,v1:127.0.0.1:6803/4040155358] [v2:127.0.0.1:6804/4040155358,v1:127.0.0.1:6805/4040155358] exists,up 7fcefe91-10b2-4f70-ac2c-7f0c3783e583 2024-03-25T23:26:24.235 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:26:24.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:26:24.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:26:24.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:26:24.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:26:24.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:26:24.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:26:24.236 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:26:24.237 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:26:24.237 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:26:24.237 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:26:24.237 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:26:24.237 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:26:24.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:26:24.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:26:24.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:26:24.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:26:24.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:26:24.345 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:26:24.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:26:24.650 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:26:24.650 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-25T23:26:24.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:26:24.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:26:24.650 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:26:24.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-03-25T23:26:24.798 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-03-25T23:26:24.798 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-03-25T23:26:24.798 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:26:24.798 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:26:24.939 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672968 2024-03-25T23:26:24.939 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672968 2024-03-25T23:26:24.940 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672968' 2024-03-25T23:26:24.940 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:26:24.940 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:26:25.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509446 2024-03-25T23:26:25.079 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509446 2024-03-25T23:26:25.079 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672968 2-64424509446' 2024-03-25T23:26:25.079 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:26:25.079 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-03-25T23:26:25.079 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:26:25.080 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:26:25.080 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-03-25T23:26:25.080 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:26:25.082 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-03-25T23:26:25.082 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-03-25T23:26:25.082 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 107374182402 2024-03-25T23:26:25.082 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:26:25.372 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-03-25T23:26:25.372 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:26:26.374 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:26:26.374 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:26:26.677 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-03-25T23:26:26.677 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:26:26.678 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672968 2024-03-25T23:26:26.678 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:26:26.679 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:26:26.680 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672968 2024-03-25T23:26:26.680 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:26:26.681 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672968 2024-03-25T23:26:26.682 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2024-03-25T23:26:26.682 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 42949672968 2024-03-25T23:26:26.682 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:26:27.020 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672968 -lt 42949672968 2024-03-25T23:26:27.020 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:26:27.021 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509446 2024-03-25T23:26:27.021 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:26:27.022 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:26:27.023 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509446 2024-03-25T23:26:27.023 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:26:27.024 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509446 2024-03-25T23:26:27.024 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-03-25T23:26:27.024 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 64424509446 2024-03-25T23:26:27.025 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:26:27.357 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509446 -lt 64424509446 2024-03-25T23:26:27.357 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:26:27.358 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:26:27.358 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:26:27.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-25T23:26:27.762 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:26:27.762 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:26:27.763 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:26:27.763 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:26:27.763 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:26:27.763 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:26:27.763 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:26:28.087 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-25T23:26:28.087 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:26:28.088 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:26:28.088 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:26:28.442 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-25T23:26:28.442 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:26:28.442 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:26:28.443 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:413: TEST_auto_repair_bluestore_tag: get_pg testpool SOMETHING 2024-03-25T23:26:28.443 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-03-25T23:26:28.443 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-25T23:26:28.443 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool SOMETHING 2024-03-25T23:26:28.443 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-25T23:26:28.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:413: TEST_auto_repair_bluestore_tag: local pgid=1.0 2024-03-25T23:26:28.777 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:414: TEST_auto_repair_bluestore_tag: get_primary testpool SOMETHING 2024-03-25T23:26:28.778 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-25T23:26:28.778 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-25T23:26:28.778 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-25T23:26:28.778 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-25T23:26:29.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:414: TEST_auto_repair_bluestore_tag: local primary=1 2024-03-25T23:26:29.120 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:415: TEST_auto_repair_bluestore_tag: echo 'Affected PG ' 1.0 ' w/ primary ' 1 2024-03-25T23:26:29.120 INFO:tasks.workunit.client.0.smithi129.stdout:Affected PG 1.0 w/ primary 1 2024-03-25T23:26:29.120 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:416: TEST_auto_repair_bluestore_tag: get_last_scrub_stamp 1.0 2024-03-25T23:26:29.121 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-25T23:26:29.121 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:26:29.121 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:26:29.121 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-25T23:26:29.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:416: TEST_auto_repair_bluestore_tag: local last_scrub_stamp=2024-03-25T23:26:10.248324+0000 2024-03-25T23:26:29.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:417: TEST_auto_repair_bluestore_tag: initiate_and_fetch_state 1 1.0 3.0 2024-03-25T23:26:29.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:322: initiate_and_fetch_state: local the_osd=osd.1 2024-03-25T23:26:29.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:323: initiate_and_fetch_state: local pgid=1.0 2024-03-25T23:26:29.431 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:324: initiate_and_fetch_state: get_last_scrub_stamp 1.0 2024-03-25T23:26:29.431 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-25T23:26:29.431 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:26:29.431 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:26:29.431 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-25T23:26:29.737 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:324: initiate_and_fetch_state: local last_scrub=2024-03-25T23:26:10.248324+0000 2024-03-25T23:26:29.737 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:326: initiate_and_fetch_state: set_config osd 1 osd_scrub_sleep 3.0 2024-03-25T23:26:29.737 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-25T23:26:29.738 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-03-25T23:26:29.738 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_scrub_sleep 2024-03-25T23:26:29.738 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=3.0 2024-03-25T23:26:29.738 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-25T23:26:29.739 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-03-25T23:26:29.739 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-25T23:26:29.739 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-25T23:26:29.740 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T23:26:29.740 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:26:29.740 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:26:29.740 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-osd.1.asok 2024-03-25T23:26:29.740 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61636/ceph-osd.1.asok config set osd_scrub_sleep 3.0 2024-03-25T23:26:29.858 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-25T23:26:29.858 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:327: initiate_and_fetch_state: set_config osd 1 osd_scrub_auto_repair true 2024-03-25T23:26:29.858 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-25T23:26:29.858 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-03-25T23:26:29.858 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_scrub_auto_repair 2024-03-25T23:26:29.858 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-25T23:26:29.859 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-03-25T23:26:29.859 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-25T23:26:29.860 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-25T23:26:29.860 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-25T23:26:29.860 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T23:26:29.860 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:26:29.860 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:26:29.860 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-osd.1.asok 2024-03-25T23:26:29.860 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61636/ceph-osd.1.asok config set osd_scrub_auto_repair true 2024-03-25T23:26:29.969 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-25T23:26:29.970 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:329: initiate_and_fetch_state: flush_pg_stats 2024-03-25T23:26:29.970 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:26:29.970 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:26:30.273 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:26:30.273 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:26:30.273 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-25T23:26:30.273 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:26:30.273 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:26:30.273 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:26:30.414 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182404 2024-03-25T23:26:30.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182404 2024-03-25T23:26:30.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182404' 2024-03-25T23:26:30.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:26:30.415 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:26:30.544 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672970 2024-03-25T23:26:30.544 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672970 2024-03-25T23:26:30.544 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182404 1-42949672970' 2024-03-25T23:26:30.544 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:26:30.544 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:26:30.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509448 2024-03-25T23:26:30.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509448 2024-03-25T23:26:30.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182404 1-42949672970 2-64424509448' 2024-03-25T23:26:30.677 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:26:30.677 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182404 2024-03-25T23:26:30.677 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:26:30.679 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:26:30.680 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182404 2024-03-25T23:26:30.680 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:26:30.680 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182404 2024-03-25T23:26:30.680 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 107374182404 2024-03-25T23:26:30.681 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182404' 2024-03-25T23:26:30.681 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:26:30.974 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182403 -lt 107374182404 2024-03-25T23:26:30.974 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:26:31.975 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:26:31.976 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:26:32.282 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182404 -lt 107374182404 2024-03-25T23:26:32.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:26:32.283 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672970 2024-03-25T23:26:32.283 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:26:32.285 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:26:32.286 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672970 2024-03-25T23:26:32.286 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:26:32.287 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 42949672970 2024-03-25T23:26:32.287 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672970 2024-03-25T23:26:32.287 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-03-25T23:26:32.288 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:26:32.618 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672970 -lt 42949672970 2024-03-25T23:26:32.618 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:26:32.619 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509448 2024-03-25T23:26:32.619 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:26:32.620 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:26:32.621 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:26:32.622 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509448 2024-03-25T23:26:32.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509448 2024-03-25T23:26:32.623 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 64424509448 2024-03-25T23:26:32.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-03-25T23:26:32.623 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:26:32.945 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509448 -lt 64424509448 2024-03-25T23:26:32.946 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:330: initiate_and_fetch_state: date --rfc-3339=ns 2024-03-25T23:26:32.946 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25 23:26:32.950565135+00:00 2024-03-25T23:26:32.947 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:333: initiate_and_fetch_state: get_asok_path osd.1 2024-03-25T23:26:32.947 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-25T23:26:32.947 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-25T23:26:32.948 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T23:26:32.948 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:26:32.948 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:26:32.948 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-osd.1.asok 2024-03-25T23:26:32.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:333: initiate_and_fetch_state: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61636/ceph-osd.1.asok schedule-deep-scrub 1.0 2024-03-25T23:26:33.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i=0 )) 2024-03-25T23:26:33.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-03-25T23:26:33.071 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-03-25T23:26:33.071 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-03-25T23:26:33.212 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-03-25T23:26:33.213 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 0 ') state now: ' '"active+clean"' 2024-03-25T23:26:33.213 INFO:tasks.workunit.client.0.smithi129.stdout:{"deep":true,"must":false,"stamp":"2024-03-11T23:24:53.061475+0000"}0 ) state now: "active+clean" 2024-03-25T23:26:33.213 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-03-25T23:26:33.213 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-03-25T23:26:33.214 INFO:tasks.workunit.client.0.smithi129.stdout:"active+clean" 2024-03-25T23:26:33.214 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 0 == 4 ']' 2024-03-25T23:26:33.214 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-03-25T23:26:33.514 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-03-25T23:26:33.514 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-03-25T23:26:33.515 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-03-25T23:26:33.515 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-03-25T23:26:33.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-03-25T23:26:33.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 1 ') state now: ' '"active+clean"' 2024-03-25T23:26:33.670 INFO:tasks.workunit.client.0.smithi129.stdout:1 ) state now: "active+clean" 2024-03-25T23:26:33.670 INFO:tasks.workunit.client.0.smithi129.stdout:"active+clean" 2024-03-25T23:26:33.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-03-25T23:26:33.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-03-25T23:26:33.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 1 == 4 ']' 2024-03-25T23:26:33.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-03-25T23:26:33.971 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-03-25T23:26:33.971 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-03-25T23:26:33.972 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-03-25T23:26:33.972 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-03-25T23:26:34.128 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-03-25T23:26:34.128 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 2 ') state now: ' '"active+clean"' 2024-03-25T23:26:34.128 INFO:tasks.workunit.client.0.smithi129.stdout:2 ) state now: "active+clean" 2024-03-25T23:26:34.128 INFO:tasks.workunit.client.0.smithi129.stdout:"active+clean" 2024-03-25T23:26:34.128 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-03-25T23:26:34.129 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-03-25T23:26:34.129 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 2 == 4 ']' 2024-03-25T23:26:34.129 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-03-25T23:26:34.429 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-03-25T23:26:34.429 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-03-25T23:26:34.430 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-03-25T23:26:34.430 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-03-25T23:26:34.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-03-25T23:26:34.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 3 ') state now: ' '"active+clean"' 2024-03-25T23:26:34.587 INFO:tasks.workunit.client.0.smithi129.stdout:3 ) state now: "active+clean" 2024-03-25T23:26:34.587 INFO:tasks.workunit.client.0.smithi129.stdout:"active+clean" 2024-03-25T23:26:34.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-03-25T23:26:34.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-03-25T23:26:34.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 3 == 4 ']' 2024-03-25T23:26:34.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-03-25T23:26:34.888 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-03-25T23:26:34.888 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-03-25T23:26:34.889 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-03-25T23:26:34.889 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-03-25T23:26:35.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-03-25T23:26:35.044 INFO:tasks.workunit.client.0.smithi129.stdout:4 ) state now: "active+clean" 2024-03-25T23:26:35.044 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 4 ') state now: ' '"active+clean"' 2024-03-25T23:26:35.044 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-03-25T23:26:35.044 INFO:tasks.workunit.client.0.smithi129.stdout:"active+clean" 2024-03-25T23:26:35.044 INFO:tasks.workunit.client.0.smithi129.stdout:loop --------> 4 2024-03-25T23:26:35.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-03-25T23:26:35.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 4 == 4 ']' 2024-03-25T23:26:35.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:350: initiate_and_fetch_state: echo 'loop --------> ' 4 2024-03-25T23:26:35.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-03-25T23:26:35.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-03-25T23:26:35.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-03-25T23:26:35.345 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-03-25T23:26:35.346 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-03-25T23:26:35.502 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-03-25T23:26:35.503 INFO:tasks.workunit.client.0.smithi129.stdout:5 ) state now: "active+clean" 2024-03-25T23:26:35.503 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 5 ') state now: ' '"active+clean"' 2024-03-25T23:26:35.503 INFO:tasks.workunit.client.0.smithi129.stdout:"active+clean" 2024-03-25T23:26:35.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-03-25T23:26:35.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-03-25T23:26:35.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 5 == 4 ']' 2024-03-25T23:26:35.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-03-25T23:26:35.804 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-03-25T23:26:35.804 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-03-25T23:26:35.805 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-03-25T23:26:35.805 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-03-25T23:26:35.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-03-25T23:26:35.965 INFO:tasks.workunit.client.0.smithi129.stdout:6 ) state now: "active+clean" 2024-03-25T23:26:35.965 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 6 ') state now: ' '"active+clean"' 2024-03-25T23:26:35.965 INFO:tasks.workunit.client.0.smithi129.stdout:"active+clean" 2024-03-25T23:26:35.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-03-25T23:26:35.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-03-25T23:26:35.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 6 == 4 ']' 2024-03-25T23:26:35.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-03-25T23:26:36.266 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-03-25T23:26:36.266 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-03-25T23:26:36.267 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-03-25T23:26:36.267 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-03-25T23:26:36.424 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-03-25T23:26:36.425 INFO:tasks.workunit.client.0.smithi129.stdout:7 ) state now: "active+clean" 2024-03-25T23:26:36.425 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 7 ') state now: ' '"active+clean"' 2024-03-25T23:26:36.425 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-03-25T23:26:36.425 INFO:tasks.workunit.client.0.smithi129.stdout:"active+clean" 2024-03-25T23:26:36.426 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-03-25T23:26:36.426 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 7 == 4 ']' 2024-03-25T23:26:36.426 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-03-25T23:26:36.726 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-03-25T23:26:36.726 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-03-25T23:26:36.727 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-03-25T23:26:36.727 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-03-25T23:26:36.886 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean+scrubbing+deep"' 2024-03-25T23:26:36.886 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 8 ') state now: ' '"active+clean+scrubbing+deep"' 2024-03-25T23:26:36.886 INFO:tasks.workunit.client.0.smithi129.stdout:8 ) state now: "active+clean+scrubbing+deep" 2024-03-25T23:26:36.886 INFO:tasks.workunit.client.0.smithi129.stdout:found scrub 2024-03-25T23:26:36.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-03-25T23:26:36.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: echo 'found scrub' 2024-03-25T23:26:36.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: return 0 2024-03-25T23:26:36.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:418: TEST_auto_repair_bluestore_tag: r=0 2024-03-25T23:26:36.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:419: TEST_auto_repair_bluestore_tag: echo 'initiate_and_fetch_state ret: ' 0 2024-03-25T23:26:36.887 INFO:tasks.workunit.client.0.smithi129.stdout:initiate_and_fetch_state ret: 0 2024-03-25T23:26:36.889 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:420: TEST_auto_repair_bluestore_tag: set_config osd td/osd-scrub-repair osd_scrub_sleep 0 2024-03-25T23:26:36.889 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-25T23:26:36.889 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=td/osd-scrub-repair 2024-03-25T23:26:36.889 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_scrub_sleep 2024-03-25T23:26:36.889 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=0 2024-03-25T23:26:36.889 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-25T23:26:36.889 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.td/osd-scrub-repair 2024-03-25T23:26:36.889 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.td/osd-scrub-repair 2024-03-25T23:26:36.890 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.td/osd-scrub-repair ']' 2024-03-25T23:26:36.890 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T23:26:36.890 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:26:36.890 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:26:36.890 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-osd.td/osd-scrub-repair.asok 2024-03-25T23:26:36.890 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61636/ceph-osd.td/osd-scrub-repair.asok config set osd_scrub_sleep 0 2024-03-25T23:26:36.983 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test == true 2024-03-25T23:26:36.983 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 1168: test: ==: unary operator expected 2024-03-25T23:26:36.983 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:421: TEST_auto_repair_bluestore_tag: '[' 0 -ne 0 ']' 2024-03-25T23:26:36.983 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:425: TEST_auto_repair_bluestore_tag: wait_end_of_scrub 1 1.0 2024-03-25T23:26:36.983 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:360: wait_end_of_scrub: local the_osd=osd.1 2024-03-25T23:26:36.983 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:361: wait_end_of_scrub: local pgid=1.0 2024-03-25T23:26:36.983 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i=0 )) 2024-03-25T23:26:36.983 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-03-25T23:26:36.983 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-03-25T23:26:36.984 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-03-25T23:26:37.134 INFO:tasks.workunit.client.0.smithi129.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-03-25T23:26:37.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-03-25T23:26:37.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-03-25T23:26:37.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-03-25T23:26:37.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 0 == 4 ']' 2024-03-25T23:26:37.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-03-25T23:26:37.436 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-03-25T23:26:37.436 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-03-25T23:26:37.437 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-03-25T23:26:37.437 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-03-25T23:26:37.591 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-03-25T23:26:37.592 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-03-25T23:26:37.592 INFO:tasks.workunit.client.0.smithi129.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-03-25T23:26:37.592 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-03-25T23:26:37.592 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 1 == 4 ']' 2024-03-25T23:26:37.593 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-03-25T23:26:37.893 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-03-25T23:26:37.893 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-03-25T23:26:37.894 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-03-25T23:26:37.894 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-03-25T23:26:38.051 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-03-25T23:26:38.051 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-03-25T23:26:38.051 INFO:tasks.workunit.client.0.smithi129.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-03-25T23:26:38.051 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-03-25T23:26:38.052 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 2 == 4 ']' 2024-03-25T23:26:38.052 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-03-25T23:26:38.353 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-03-25T23:26:38.353 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-03-25T23:26:38.354 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-03-25T23:26:38.354 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-03-25T23:26:38.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-03-25T23:26:38.510 INFO:tasks.workunit.client.0.smithi129.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-03-25T23:26:38.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-03-25T23:26:38.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-03-25T23:26:38.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 3 == 4 ']' 2024-03-25T23:26:38.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-03-25T23:26:38.811 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-03-25T23:26:38.811 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-03-25T23:26:38.812 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-03-25T23:26:38.812 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-03-25T23:26:38.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-03-25T23:26:38.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-03-25T23:26:38.972 INFO:tasks.workunit.client.0.smithi129.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-03-25T23:26:38.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-03-25T23:26:38.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 4 == 4 ']' 2024-03-25T23:26:38.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: flush_pg_stats 2024-03-25T23:26:38.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:26:38.973 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:26:39.290 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:26:39.290 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:26:39.290 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-25T23:26:39.290 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:26:39.290 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:26:39.291 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:26:39.466 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182407 2024-03-25T23:26:39.466 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182407 2024-03-25T23:26:39.466 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182407' 2024-03-25T23:26:39.466 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:26:39.466 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:26:39.612 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672973 2024-03-25T23:26:39.612 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672973 2024-03-25T23:26:39.612 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182407 1-42949672973' 2024-03-25T23:26:39.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:26:39.613 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:26:39.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509450 2024-03-25T23:26:39.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509450 2024-03-25T23:26:39.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182407 1-42949672973 2-64424509450' 2024-03-25T23:26:39.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:26:39.758 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182407 2024-03-25T23:26:39.758 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:26:39.759 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:26:39.760 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182407 2024-03-25T23:26:39.760 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:26:39.761 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 107374182407 2024-03-25T23:26:39.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182407 2024-03-25T23:26:39.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182407' 2024-03-25T23:26:39.761 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:26:40.074 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182407 -lt 107374182407 2024-03-25T23:26:40.074 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:26:40.075 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672973 2024-03-25T23:26:40.075 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:26:40.077 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:26:40.077 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672973 2024-03-25T23:26:40.077 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:26:40.079 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672973 2024-03-25T23:26:40.079 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-03-25T23:26:40.079 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 42949672973 2024-03-25T23:26:40.079 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:26:40.411 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672973 -lt 42949672973 2024-03-25T23:26:40.412 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:26:40.412 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:26:40.412 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509450 2024-03-25T23:26:40.414 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:26:40.414 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509450 2024-03-25T23:26:40.414 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:26:40.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509450 2024-03-25T23:26:40.416 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 64424509450 2024-03-25T23:26:40.416 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-03-25T23:26:40.416 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:26:40.745 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509449 -lt 64424509450 2024-03-25T23:26:40.745 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:26:41.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:26:41.747 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:26:42.051 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509451 -lt 64424509450 2024-03-25T23:26:42.052 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-03-25T23:26:42.353 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-03-25T23:26:42.353 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-03-25T23:26:42.354 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-03-25T23:26:42.354 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-03-25T23:26:42.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean"' 2024-03-25T23:26:42.511 INFO:tasks.workunit.client.0.smithi129.stdout:wait-scrub-end state now: "active+clean" 2024-03-25T23:26:42.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean"' 2024-03-25T23:26:42.512 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2024-03-25T23:26:42.512 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: break 2024-03-25T23:26:42.512 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:373: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2024-03-25T23:26:42.512 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:378: wait_end_of_scrub: return 0 2024-03-25T23:26:42.512 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:426: TEST_auto_repair_bluestore_tag: ceph pg dump pgs 2024-03-25T23:26:42.783 INFO:tasks.workunit.client.0.smithi129.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-25T23:26:42.784 INFO:tasks.workunit.client.0.smithi129.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean+scrubbing+deep 2024-03-25T23:26:36.782806+0000 21'1 26:44 [1,0] 1 [1,0] 1 0'0 2024-03-11T23:24:53.061475+0000 0'0 2024-03-11T23:24:53.061475+0000 0 0 deep scrubbing for 0s 0 0 2024-03-25T23:26:42.784 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-25T23:26:42.784 INFO:tasks.workunit.client.0.smithi129.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-25T23:26:42.784 INFO:tasks.workunit.client.0.smithi129.stderr:dumped pgs 2024-03-25T23:26:42.800 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-03-25T23:26:42.800 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-25T23:26:42.800 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-25T23:26:42.800 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-25T23:26:42.801 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-25T23:26:42.801 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-25T23:26:42.801 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-25T23:26:42.801 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-25T23:26:43.119 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-25T23:26:43.119 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-25T23:26:43.119 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-25T23:26:43.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-25T23:26:43.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:26:43.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:26:43.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-25T23:26:43.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:26:43.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-25T23:26:43.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:26:43.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:26:43.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-25T23:26:43.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:26:43.454 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-25T23:26:43.454 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:26:43.454 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:26:43.454 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:26:43.454 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:26:43.454 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:26:43.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:26:43.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-25T23:26:43.762 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:26:43.762 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:26:43.762 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-25T23:26:43.762 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:26:43.762 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-25T23:26:43.762 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-03-25T23:26:43.985 INFO:tasks.workunit.client.0.smithi129.stdout:_ 2024-03-25T23:26:43.985 INFO:tasks.workunit.client.0.smithi129.stdout:snapset 2024-03-25T23:26:44.286 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-25T23:26:44.287 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:26:44.287 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:26:44.287 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-25T23:26:44.287 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:26:44.287 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-25T23:26:44.287 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:26:44.287 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:26:44.287 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:26:44.287 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:26:44.287 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-25T23:26:44.287 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-25T23:26:44.287 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:26:44.287 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:26:44.287 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:26:44.288 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:26:44.288 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:26:44.288 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:26:44.288 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:26:44.288 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:26:44.288 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:26:44.288 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:26:44.289 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:26:44.289 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:26:44.289 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:26:44.289 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:26:44.289 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:26:44.289 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:26:44.289 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:26:44.289 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:26:44.289 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:26:44.290 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-03-25T23:26:44.290 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-25T23:26:44.290 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-25T23:26:44.290 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.0 2024-03-25T23:26:44.290 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-25T23:26:44.291 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-25T23:26:44.291 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-25T23:26:44.291 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:26:44.291 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:26:44.291 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:26:44.332 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:26:44.332+0000 7f24759687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:26:44.342 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:26:44.344+0000 7f24759687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:26:44.346 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:26:44.344+0000 7f24759687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:26:44.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-25T23:26:44.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:26:44.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T23:26:44.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:26:44.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:26:44.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:26:44.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:26:44.608 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-25T23:26:44.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:26:44.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:26:44.884 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:26:44.884+0000 7f24759687c0 -1 Falling back to public interface 2024-03-25T23:26:44.946 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:26:45.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:26:45.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:26:45.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:26:45.948 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:26:45.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:26:45.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:26:46.269 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:26:47.271 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:26:47.271 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:26:47.271 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:26:47.271 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:26:47.272 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:26:47.272 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:26:47.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:26:47.857 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:26:47.856+0000 7f24759687c0 -1 osd.0 26 log_to_monitors true 2024-03-25T23:26:48.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:26:48.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:26:48.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:26:48.598 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:26:48.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:26:48.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:26:48.919 INFO:tasks.workunit.client.0.smithi129.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/4112437986,v1:127.0.0.1:6803/4112437986] [v2:127.0.0.1:6804/4112437986,v1:127.0.0.1:6805/4112437986] exists,up 7fcefe91-10b2-4f70-ac2c-7f0c3783e583 2024-03-25T23:26:48.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:26:48.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:26:48.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:26:48.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:26:48.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:26:48.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:26:48.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:26:48.921 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:26:48.921 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:26:48.921 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:26:48.921 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:26:48.921 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:26:48.921 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:26:49.030 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:26:49.030 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:26:49.030 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:26:49.030 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:26:49.030 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:26:49.030 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:26:49.381 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:26:49.382 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:26:49.382 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-25T23:26:49.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:26:49.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:26:49.382 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:26:49.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-03-25T23:26:49.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-03-25T23:26:49.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882' 2024-03-25T23:26:49.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:26:49.579 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:26:49.719 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672976 2024-03-25T23:26:49.719 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672976 2024-03-25T23:26:49.719 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672976' 2024-03-25T23:26:49.719 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:26:49.719 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:26:49.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-03-25T23:26:49.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-03-25T23:26:49.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672976 2-64424509453' 2024-03-25T23:26:49.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:26:49.861 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018882 2024-03-25T23:26:49.861 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:26:49.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:26:49.862 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018882 2024-03-25T23:26:49.862 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:26:49.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-03-25T23:26:49.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-03-25T23:26:49.863 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 128849018882 2024-03-25T23:26:49.864 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:26:50.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-03-25T23:26:50.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:26:50.169 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672976 2024-03-25T23:26:50.169 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:26:50.170 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:26:50.171 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672976 2024-03-25T23:26:50.172 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:26:50.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672976 2024-03-25T23:26:50.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-03-25T23:26:50.173 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 42949672976 2024-03-25T23:26:50.173 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:26:50.520 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672975 -lt 42949672976 2024-03-25T23:26:50.520 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:26:51.523 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:26:51.524 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:26:51.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672976 2024-03-25T23:26:51.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:26:51.834 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-03-25T23:26:51.834 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:26:51.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:26:51.836 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:26:51.836 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-03-25T23:26:51.837 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-03-25T23:26:51.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-03-25T23:26:51.838 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 64424509453 2024-03-25T23:26:51.838 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:26:52.159 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509454 -lt 64424509453 2024-03-25T23:26:52.159 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:26:52.160 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:26:52.160 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:26:52.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-25T23:26:52.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:26:52.546 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:26:52.546 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:26:52.547 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:26:52.547 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:26:52.547 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:26:52.547 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:26:52.851 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-25T23:26:52.852 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:26:52.852 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:26:52.852 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:26:53.214 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-25T23:26:53.214 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:26:53.214 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:26:53.214 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:431: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-03-25T23:26:53.214 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-25T23:26:53.215 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-25T23:26:53.215 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-25T23:26:53.215 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-25T23:26:53.216 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-25T23:26:53.216 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-25T23:26:53.216 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-25T23:26:53.514 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-25T23:26:53.514 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-25T23:26:53.515 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-25T23:26:53.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:431: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-03-25T23:26:53.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:26:53.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:26:53.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-25T23:26:53.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:26:53.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-03-25T23:26:53.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:26:53.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:26:53.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-25T23:26:53.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:26:53.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-25T23:26:53.861 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:26:53.861 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:26:53.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:26:53.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:26:53.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:26:54.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:26:54.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-03-25T23:26:54.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:26:54.170 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:26:54.170 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-25T23:26:54.170 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:26:54.170 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-25T23:26:54.170 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-03-25T23:26:54.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-25T23:26:54.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:26:54.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:26:54.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-25T23:26:54.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:26:54.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-25T23:26:54.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:26:54.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:26:54.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:26:54.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:26:54.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-25T23:26:54.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-25T23:26:54.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:26:54.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:26:54.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:26:54.586 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:26:54.586 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:26:54.586 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:26:54.586 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:26:54.587 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:26:54.587 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:26:54.588 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:26:54.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:26:54.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:26:54.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:26:54.589 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:26:54.589 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:26:54.589 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:26:54.589 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:26:54.589 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:26:54.589 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:26:54.589 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-03-25T23:26:54.589 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-25T23:26:54.589 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-25T23:26:54.589 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.0 2024-03-25T23:26:54.589 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-25T23:26:54.589 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-25T23:26:54.590 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-25T23:26:54.594 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:26:54.594 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:26:54.594 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:26:54.625 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:26:54.624+0000 7f2b0ea6c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:26:54.632 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:26:54.632+0000 7f2b0ea6c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:26:54.634 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:26:54.632+0000 7f2b0ea6c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:26:54.906 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-25T23:26:54.906 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:26:54.906 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T23:26:54.906 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:26:54.906 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:26:54.906 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:26:54.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:26:54.907 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-25T23:26:54.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:26:54.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:26:55.168 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:26:55.167+0000 7f2b0ea6c7c0 -1 Falling back to public interface 2024-03-25T23:26:55.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:26:56.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:26:56.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:26:56.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:26:56.230 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:26:56.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:26:56.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:26:56.539 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:26:57.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:26:57.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:26:57.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:26:57.541 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:26:57.542 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:26:57.542 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:26:57.866 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:26:58.301 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:26:58.299+0000 7f2b0ea6c7c0 -1 osd.0 31 log_to_monitors true 2024-03-25T23:26:58.867 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:26:58.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:26:58.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:26:58.868 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:26:58.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:26:58.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:26:59.227 INFO:tasks.workunit.client.0.smithi129.stderr:osd.0 up in weight 1 up_from 35 up_thru 0 down_at 32 last_clean_interval [30,31) [v2:127.0.0.1:6802/1804899879,v1:127.0.0.1:6803/1804899879] [v2:127.0.0.1:6804/1804899879,v1:127.0.0.1:6805/1804899879] exists,up 7fcefe91-10b2-4f70-ac2c-7f0c3783e583 2024-03-25T23:26:59.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:26:59.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:26:59.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:26:59.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:26:59.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:26:59.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:26:59.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:26:59.228 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:26:59.228 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:26:59.228 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:26:59.229 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:26:59.229 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:26:59.229 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:26:59.340 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:26:59.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:26:59.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:26:59.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:26:59.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:26:59.341 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:26:59.658 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:26:59.659 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:26:59.659 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-25T23:26:59.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:26:59.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:26:59.659 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:26:59.876 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-03-25T23:26:59.876 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-03-25T23:26:59.876 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362' 2024-03-25T23:26:59.876 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:26:59.877 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:27:00.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672979 2024-03-25T23:27:00.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672979 2024-03-25T23:27:00.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-42949672979' 2024-03-25T23:27:00.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:27:00.078 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:27:00.275 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509457 2024-03-25T23:27:00.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509457 2024-03-25T23:27:00.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-42949672979 2-64424509457' 2024-03-25T23:27:00.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:27:00.276 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855362 2024-03-25T23:27:00.276 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:27:00.277 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:27:00.278 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:27:00.278 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855362 2024-03-25T23:27:00.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-03-25T23:27:00.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2024-03-25T23:27:00.280 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 150323855362 2024-03-25T23:27:00.280 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:27:00.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 150323855362 2024-03-25T23:27:00.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:27:01.624 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:27:01.625 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:27:01.942 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-03-25T23:27:01.942 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:27:01.943 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672979 2024-03-25T23:27:01.943 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:27:01.945 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:27:01.946 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:27:01.946 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672979 2024-03-25T23:27:01.947 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672979 2024-03-25T23:27:01.947 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-03-25T23:27:01.947 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 42949672979 2024-03-25T23:27:01.948 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:27:02.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672979 -lt 42949672979 2024-03-25T23:27:02.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:27:02.237 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509457 2024-03-25T23:27:02.237 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:27:02.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:27:02.239 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509457 2024-03-25T23:27:02.239 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:27:02.241 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509457 2024-03-25T23:27:02.241 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-03-25T23:27:02.241 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 64424509457 2024-03-25T23:27:02.242 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:27:02.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509457 -lt 64424509457 2024-03-25T23:27:02.554 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:27:02.554 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:27:02.554 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:27:02.954 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-25T23:27:02.954 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:27:02.955 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:27:02.955 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:27:02.955 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:27:02.955 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:27:02.955 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:27:02.956 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:27:03.295 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-25T23:27:03.295 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:27:03.296 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:27:03.296 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:27:03.652 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-03-25T23:27:03.652 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-03-25T23:27:03.652 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-25T23:27:03.652 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-03-25T23:27:03.652 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:27:03.652 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-25T23:27:03.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:27:03.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:27:03.754 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:27:03.754 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:27:03.754 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:27:03.754 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:27:03.755 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:27:03.755 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:27:04.054 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-25T23:27:04.055 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:27:04.055 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:27:04.055 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:27:04.416 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-25T23:27:04.417 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:27:04.417 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:27:04.417 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:432: TEST_auto_repair_bluestore_tag: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-25T23:27:04.417 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:433: TEST_auto_repair_bluestore_tag: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-03-25T23:27:04.418 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:26:39.792+0000 7f6a8b917640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-03-25T23:27:04.419 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:26:39.792+0000 7f6a8b917640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-03-25T23:27:04.419 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:26:39.792+0000 7f6a8b917640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-03-25T23:27:04.419 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:26:39.792+0000 7f6a8b917640 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-03-25T23:27:04.419 INFO:tasks.workunit.client.0.smithi129.stdout:2024-03-25T23:26:39.792+0000 7f6a8b917640 19 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-25T23:27:04.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-25T23:27:04.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-25T23:27:04.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T23:27:04.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-25T23:27:04.420 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:27:04.420 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:27:04.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:27:04.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:27:04.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:27:04.728 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:27:04.729 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T23:27:04.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T23:27:04.731 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T23:27:04.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T23:27:04.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T23:27:04.732 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T23:27:04.733 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:27:04.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T23:27:04.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T23:27:04.734 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:27:04.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T23:27:04.736 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T23:27:04.737 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-25T23:27:04.747 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T23:27:04.747 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:27:04.748 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:27:04.748 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-25T23:27:04.750 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T23:27:04.750 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T23:27:04.750 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-25T23:27:04.750 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-25T23:27:04.750 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-25T23:27:04.750 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-25T23:27:04.750 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-25T23:27:04.751 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T23:27:04.751 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-25T23:27:04.751 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:27:04.751 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:27:04.751 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:27:04.751 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:27:04.751 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:27:04.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:27:04.753 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T23:27:04.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T23:27:04.754 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T23:27:04.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T23:27:04.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T23:27:04.755 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T23:27:04.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:27:04.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T23:27:04.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T23:27:04.757 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:27:04.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T23:27:04.759 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T23:27:04.759 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-25T23:27:04.761 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T23:27:04.761 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:27:04.761 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:27:04.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-25T23:27:04.762 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T23:27:04.762 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T23:27:04.762 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-25T23:27:04.764 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-25T23:27:04.764 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:27:04.764 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:27:04.764 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61636 2024-03-25T23:27:04.767 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-25T23:27:04.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-25T23:27:04.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-25T23:27:04.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-25T23:27:04.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_erasure_coded_appends td/osd-scrub-repair 2024-03-25T23:27:04.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:309: TEST_auto_repair_erasure_coded_appends: auto_repair_erasure_coded td/osd-scrub-repair false 2024-03-25T23:27:04.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-03-25T23:27:04.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: auto_repair_erasure_coded: local allow_overwrites=false 2024-03-25T23:27:04.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: auto_repair_erasure_coded: local poolname=ecpool 2024-03-25T23:27:04.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2024-03-25T23:27:04.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-25T23:27:04.768 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-25T23:27:04.768 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-25T23:27:04.768 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-25T23:27:04.768 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-25T23:27:04.768 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-25T23:27:05.241 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-25T23:27:05.241 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:27:05.241 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:27:05.242 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:27:05.242 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:27:05.242 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:27:05.242 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:27:05.245 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-25T23:27:05.462 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-25T23:27:05.463 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-25T23:27:05.463 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T23:27:05.463 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T23:27:05.463 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-25T23:27:05.464 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-25T23:27:05.464 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T23:27:05.464 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T23:27:05.464 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T23:27:05.465 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T23:27:05.465 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:27:05.465 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:27:05.465 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-25T23:27:05.465 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T23:27:05.465 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get fsid 2024-03-25T23:27:05.581 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-25T23:27:05.582 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T23:27:05.582 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T23:27:05.582 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-25T23:27:05.583 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-25T23:27:05.583 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T23:27:05.583 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T23:27:05.583 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T23:27:05.583 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T23:27:05.583 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:27:05.583 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:27:05.583 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-25T23:27:05.584 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T23:27:05.584 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get mon_host 2024-03-25T23:27:05.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:274: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2024-03-25T23:27:05.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-25T23:27:05.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-25T23:27:05.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-25T23:27:05.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-25T23:27:05.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-25T23:27:05.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-25T23:27:06.603 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-25T23:27:06.603 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:27:06.603 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:27:06.603 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:27:06.603 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:27:06.603 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:27:06.603 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:27:06.604 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T23:27:06.605 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T23:27:06.639 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-25T23:27:06.640 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: seq 0 2 2024-03-25T23:27:06.647 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-03-25T23:27:06.647 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:27:06.647 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:27:06.647 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:27:06.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-25T23:27:06.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:27:06.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-25T23:27:06.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:27:06.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:27:06.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:27:06.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:27:06.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-25T23:27:06.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-25T23:27:06.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:27:06.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:27:06.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:27:06.650 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:27:06.650 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:27:06.650 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:27:06.650 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:27:06.650 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:27:06.651 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:27:06.651 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:27:06.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:27:06.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:27:06.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:27:06.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:27:06.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:27:06.652 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:27:06.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:27:06.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:27:06.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:27:06.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:27:06.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:27:06.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-25T23:27:06.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-25T23:27:06.653 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:27:06.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8ec4a3fe-c190-4ab2-a034-bb6b22817c84 2024-03-25T23:27:06.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8ec4a3fe-c190-4ab2-a034-bb6b22817c84' 2024-03-25T23:27:06.654 INFO:tasks.workunit.client.0.smithi129.stdout:add osd0 8ec4a3fe-c190-4ab2-a034-bb6b22817c84 2024-03-25T23:27:06.654 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:27:06.663 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBKCAJmdQirJxAAln+TFhFjx12Zg59CJr679w== 2024-03-25T23:27:06.663 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBKCAJmdQirJxAAln+TFhFjx12Zg59CJr679w=="}' 2024-03-25T23:27:06.663 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8ec4a3fe-c190-4ab2-a034-bb6b22817c84 -i td/osd-scrub-repair/0/new.json 2024-03-25T23:27:06.931 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:27:06.947 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-25T23:27:06.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBKCAJmdQirJxAAln+TFhFjx12Zg59CJr679w== --osd-uuid 8ec4a3fe-c190-4ab2-a034-bb6b22817c84 2024-03-25T23:27:06.985 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:27:06.983+0000 7f2d4b3c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:27:06.999 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:27:06.999+0000 7f2d4b3c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:27:07.000 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:27:06.999+0000 7f2d4b3c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:27:07.001 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:27:06.999+0000 7f2d4b3c57c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-25T23:27:07.001 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:27:06.999+0000 7f2d4b3c57c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-25T23:27:07.001 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:27:06.999+0000 7f2d4b3c57c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-25T23:27:07.091 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:27:07.091+0000 7f2d4b3c57c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-25T23:27:13.396 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-25T23:27:13.397 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:27:13.398 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd0 key to auth repository 2024-03-25T23:27:13.399 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-25T23:27:13.399 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:27:13.787 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-25T23:27:13.787 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.0 2024-03-25T23:27:13.788 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:27:13.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:27:13.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:27:13.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:27:13.824 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:27:13.823+0000 7fbd050b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:27:13.858 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:27:13.855+0000 7fbd050b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:27:13.859 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:27:13.859+0000 7fbd050b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:27:14.131 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-25T23:27:14.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:27:14.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T23:27:14.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:27:14.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:27:14.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:27:14.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:27:14.132 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:27:14.133 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:27:14.133 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:27:14.421 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:27:14.419+0000 7fbd050b47c0 -1 Falling back to public interface 2024-03-25T23:27:14.475 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:27:15.476 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:27:15.477 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:27:15.477 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:27:15.478 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:27:15.478 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:27:15.478 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:27:15.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:27:16.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:27:16.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:27:16.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:27:16.780 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:27:16.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:27:16.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:27:16.983 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:27:16.983+0000 7fbd050b47c0 -1 osd.0 0 log_to_monitors true 2024-03-25T23:27:17.099 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:27:18.101 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:27:18.101 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:27:18.101 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:27:18.101 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:27:18.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:27:18.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:27:18.425 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:27:19.426 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:27:19.427 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:27:19.427 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:27:19.428 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:27:19.428 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:27:19.428 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:27:19.851 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:27:20.853 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:27:20.853 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:27:20.853 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:27:20.854 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:27:20.854 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:27:20.854 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:27:21.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:27:22.171 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:27:22.171 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:27:22.171 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:27:22.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:27:22.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:27:22.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:27:22.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:27:23.492 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:27:23.492 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:27:23.492 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:27:23.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:27:23.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:27:23.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:27:23.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:27:24.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:27:24.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:27:24.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:27:24.838 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:27:24.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:27:24.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:27:25.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:27:26.159 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:27:26.159 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:27:26.160 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:27:26.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:27:26.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:27:26.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:27:26.478 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:27:27.480 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:27:27.480 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:27:27.480 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:27:27.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:27:27.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:27:27.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:27:27.859 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:27:28.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:27:28.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:27:28.861 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:27:28.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:27:28.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:27:28.862 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:27:29.174 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:27:30.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:27:30.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:27:30.176 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:27:30.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:27:30.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:27:30.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:27:30.488 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:27:31.490 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:27:31.490 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:27:31.491 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:27:31.491 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:27:31.491 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:27:31.491 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:27:31.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:27:32.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:27:32.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:27:32.917 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:27:32.918 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:27:32.918 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:27:32.918 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:27:33.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:27:34.437 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:27:34.437 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:27:34.437 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:27:34.438 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:27:34.438 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:27:34.439 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:27:34.942 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:27:35.944 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:27:35.944 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:27:35.945 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:27:35.945 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:27:35.945 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:27:35.945 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:27:36.362 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:27:37.364 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:27:37.364 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:27:37.364 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:27:37.365 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:27:37.365 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:27:37.365 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:27:38.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:27:39.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:27:39.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:27:39.177 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:27:39.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:27:39.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:27:39.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:27:39.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:27:40.662 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:27:40.662 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:27:40.662 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:27:40.663 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:27:40.663 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:27:40.663 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:27:41.126 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:27:42.128 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:27:42.128 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:27:42.129 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-25T23:27:42.129 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-25T23:27:42.130 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:27:42.130 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:27:42.448 INFO:tasks.workunit.client.0.smithi129.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3573635045,v1:127.0.0.1:6803/3573635045] [v2:127.0.0.1:6804/3573635045,v1:127.0.0.1:6805/3573635045] exists,up 8ec4a3fe-c190-4ab2-a034-bb6b22817c84 2024-03-25T23:27:42.451 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:27:42.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:27:42.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:27:42.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-03-25T23:27:42.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:27:42.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:27:42.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:27:42.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-25T23:27:42.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:27:42.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:27:42.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:27:42.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:27:42.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:27:42.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:27:42.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-25T23:27:42.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-25T23:27:42.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:27:42.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:27:42.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:27:42.453 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:27:42.453 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:27:42.453 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:27:42.453 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:27:42.453 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:27:42.453 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:27:42.453 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:27:42.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:27:42.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:27:42.454 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:27:42.454 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:27:42.454 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:27:42.454 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:27:42.454 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:27:42.454 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:27:42.454 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:27:42.454 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:27:42.454 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:27:42.454 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-25T23:27:42.454 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-25T23:27:42.455 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:27:42.455 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1925eaee-ecba-41a8-9e4f-46b8035d3274 2024-03-25T23:27:42.455 INFO:tasks.workunit.client.0.smithi129.stdout:add osd1 1925eaee-ecba-41a8-9e4f-46b8035d3274 2024-03-25T23:27:42.455 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 1925eaee-ecba-41a8-9e4f-46b8035d3274' 2024-03-25T23:27:42.456 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:27:42.472 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBuCAJmwOdHHBAAD0VjFENvDzEZuM/vjJStog== 2024-03-25T23:27:42.472 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBuCAJmwOdHHBAAD0VjFENvDzEZuM/vjJStog=="}' 2024-03-25T23:27:42.472 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1925eaee-ecba-41a8-9e4f-46b8035d3274 -i td/osd-scrub-repair/1/new.json 2024-03-25T23:27:42.850 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:27:42.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-25T23:27:42.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBuCAJmwOdHHBAAD0VjFENvDzEZuM/vjJStog== --osd-uuid 1925eaee-ecba-41a8-9e4f-46b8035d3274 2024-03-25T23:27:42.908 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:27:42.906+0000 7fc248af37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:27:42.952 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:27:42.950+0000 7fc248af37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:27:42.954 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:27:42.950+0000 7fc248af37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:27:42.955 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:27:42.954+0000 7fc248af37c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-25T23:27:42.955 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:27:42.954+0000 7fc248af37c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-25T23:27:42.955 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:27:42.954+0000 7fc248af37c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-25T23:27:43.033 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:27:43.034+0000 7fc248af37c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-25T23:27:50.001 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-25T23:27:50.001 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:27:50.002 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-25T23:27:50.002 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd1 key to auth repository 2024-03-25T23:27:50.002 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:27:50.388 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.1 2024-03-25T23:27:50.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-25T23:27:50.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:27:50.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:27:50.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:27:50.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:27:50.423 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:27:50.422+0000 7f9aabf0d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:27:50.461 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:27:50.458+0000 7f9aabf0d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:27:50.462 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:27:50.462+0000 7f9aabf0d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:27:50.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-25T23:27:50.741 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:27:50.741 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T23:27:50.741 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:27:50.741 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:27:50.741 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:27:50.742 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:27:50.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:27:50.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:27:50.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:27:51.017 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:27:51.014+0000 7f9aabf0d7c0 -1 Falling back to public interface 2024-03-25T23:27:51.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:27:52.083 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:27:52.083 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:27:52.083 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:27:52.084 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:27:52.084 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:27:52.084 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:27:52.403 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:27:53.403 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:27:53.404 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:27:53.404 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:27:53.404 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:27:53.405 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:27:53.405 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:27:53.750 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:27:53.746+0000 7f9aabf0d7c0 -1 osd.1 0 log_to_monitors true 2024-03-25T23:27:53.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:27:54.769 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:27:54.769 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:27:54.770 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:27:54.770 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:27:54.770 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:27:54.770 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:27:55.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:27:56.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:27:56.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:27:56.179 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:27:56.180 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:27:56.180 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:27:56.180 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:27:56.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:27:57.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:27:57.566 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:27:57.566 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:27:57.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:27:57.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:27:57.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:27:57.969 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:27:58.971 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:27:58.971 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:27:58.971 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:27:58.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:27:58.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:27:58.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:27:59.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:28:00.300 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:28:00.300 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:28:00.300 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:28:00.301 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:28:00.301 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:28:00.301 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:28:00.622 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:28:01.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:28:01.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:28:01.624 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:28:01.624 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:28:01.624 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:28:01.625 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:28:02.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:28:03.098 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:28:03.098 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:28:03.098 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:28:03.099 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:28:03.099 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:28:03.099 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:28:03.422 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:28:04.424 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:28:04.424 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:28:04.424 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:28:04.424 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:28:04.425 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:28:04.425 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:28:04.888 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:28:05.890 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:28:05.890 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:28:05.890 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:28:05.891 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:28:05.891 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:28:05.891 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:28:06.255 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:28:07.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:28:07.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:28:07.257 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:28:07.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:28:07.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:28:07.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:28:07.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:28:08.582 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:28:08.582 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:28:08.583 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:28:08.583 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:28:08.583 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:28:08.583 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:28:09.221 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:28:10.222 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:28:10.222 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:28:10.223 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:28:10.223 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:28:10.223 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:28:10.223 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:28:11.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:28:12.242 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:28:12.242 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:28:12.243 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:28:12.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:28:12.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:28:12.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:28:12.619 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:28:13.621 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:28:13.621 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:28:13.621 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:28:13.622 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:28:13.622 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:28:13.622 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:28:14.439 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:28:15.441 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:28:15.441 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:28:15.441 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:28:15.442 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:28:15.442 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:28:15.442 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:28:16.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:28:17.190 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:28:17.190 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:28:17.191 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:28:17.191 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:28:17.191 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:28:17.191 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:28:18.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:28:19.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:28:19.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:28:19.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:28:19.069 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:28:19.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:28:19.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:28:19.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:28:20.574 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:28:20.574 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:28:20.575 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-25T23:28:20.575 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-25T23:28:20.575 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:28:20.575 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:28:20.908 INFO:tasks.workunit.client.0.smithi129.stdout:osd.1 up in weight 1 up_from 16 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3689996859,v1:127.0.0.1:6811/3689996859] [v2:127.0.0.1:6812/3689996859,v1:127.0.0.1:6813/3689996859] exists,up 1925eaee-ecba-41a8-9e4f-46b8035d3274 2024-03-25T23:28:20.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:28:20.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:28:20.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:28:20.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-03-25T23:28:20.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:28:20.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:28:20.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:28:20.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-25T23:28:20.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:28:20.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-25T23:28:20.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:28:20.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:28:20.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:28:20.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:28:20.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-25T23:28:20.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-25T23:28:20.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:28:20.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:28:20.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:28:20.911 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:28:20.911 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:28:20.911 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:28:20.912 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:28:20.912 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:28:20.912 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:28:20.912 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:28:20.913 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:28:20.913 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:28:20.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:28:20.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:28:20.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:28:20.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:28:20.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:28:20.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:28:20.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:28:20.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:28:20.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:28:20.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-25T23:28:20.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-25T23:28:20.914 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:28:20.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4c4afab9-e887-4776-85f3-b07114ad18ab 2024-03-25T23:28:20.915 INFO:tasks.workunit.client.0.smithi129.stdout:add osd2 4c4afab9-e887-4776-85f3-b07114ad18ab 2024-03-25T23:28:20.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 4c4afab9-e887-4776-85f3-b07114ad18ab' 2024-03-25T23:28:20.915 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:28:20.938 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCUCAJmrAUNOBAA0Rfi6FQ9mVHC2nkPUNbAxQ== 2024-03-25T23:28:20.938 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCUCAJmrAUNOBAA0Rfi6FQ9mVHC2nkPUNbAxQ=="}' 2024-03-25T23:28:20.938 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4c4afab9-e887-4776-85f3-b07114ad18ab -i td/osd-scrub-repair/2/new.json 2024-03-25T23:28:21.419 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:28:21.438 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-25T23:28:21.440 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCUCAJmrAUNOBAA0Rfi6FQ9mVHC2nkPUNbAxQ== --osd-uuid 4c4afab9-e887-4776-85f3-b07114ad18ab 2024-03-25T23:28:21.479 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:28:21.481+0000 7f91ca8ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:28:21.512 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:28:21.513+0000 7f91ca8ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:28:21.514 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:28:21.513+0000 7f91ca8ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:28:21.514 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:28:21.513+0000 7f91ca8ed7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-25T23:28:21.514 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:28:21.513+0000 7f91ca8ed7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-25T23:28:21.514 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:28:21.513+0000 7f91ca8ed7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-25T23:28:21.584 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:28:21.585+0000 7f91ca8ed7c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-25T23:28:28.417 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-25T23:28:28.417 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:28:28.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-25T23:28:28.419 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd2 key to auth repository 2024-03-25T23:28:28.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:28:28.812 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.2 2024-03-25T23:28:28.813 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-25T23:28:28.813 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:28:28.813 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:28:28.813 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:28:28.813 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:28:28.848 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:28:28.849+0000 7f9b4bb2a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:28:28.873 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:28:28.873+0000 7f9b4bb2a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:28:28.874 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:28:28.877+0000 7f9b4bb2a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:28:29.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-25T23:28:29.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:28:29.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-25T23:28:29.173 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:28:29.174 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:28:29.174 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:28:29.174 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:28:29.174 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:28:29.174 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:28:29.174 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:28:29.429 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:28:29.429+0000 7f9b4bb2a7c0 -1 Falling back to public interface 2024-03-25T23:28:29.525 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:28:30.526 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:28:30.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:28:30.527 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:28:30.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:28:30.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:28:30.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:28:30.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:28:31.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:28:31.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:28:31.835 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:28:31.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:28:31.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:28:31.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:28:32.258 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:28:32.587 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:28:32.589+0000 7f9b4bb2a7c0 -1 osd.2 0 log_to_monitors true 2024-03-25T23:28:33.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:28:33.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:28:33.260 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:28:33.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:28:33.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:28:33.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:28:33.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:28:34.633 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:28:34.633 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:28:34.633 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:28:34.634 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:28:34.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:28:34.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:28:34.950 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:28:35.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:28:35.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:28:35.953 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:28:35.953 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:28:35.953 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:28:35.953 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:28:36.321 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:28:37.323 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:28:37.324 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:28:37.324 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:28:37.324 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:28:37.325 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:28:37.325 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:28:37.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:28:38.762 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:28:38.762 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:28:38.763 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:28:38.763 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:28:38.763 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:28:38.763 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:28:39.087 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:28:40.089 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:28:40.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:28:40.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:28:40.090 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:28:40.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:28:40.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:28:40.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:28:41.610 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:28:41.610 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:28:41.611 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:28:41.611 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:28:41.611 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:28:41.611 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:28:42.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:28:43.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:28:43.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:28:43.048 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:28:43.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:28:43.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:28:43.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:28:43.625 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:28:44.628 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:28:44.628 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:28:44.628 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:28:44.628 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:28:44.628 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:28:44.629 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:28:45.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:28:46.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:28:46.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:28:46.047 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:28:46.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:28:46.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:28:46.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:28:46.423 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:28:47.425 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:28:47.425 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:28:47.426 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:28:47.426 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:28:47.426 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:28:47.426 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:28:47.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:28:48.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:28:48.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:28:48.781 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:28:48.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:28:48.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:28:48.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:28:49.699 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:28:50.701 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:28:50.701 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:28:50.701 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:28:50.702 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:28:50.702 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:28:50.702 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:28:51.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:28:52.408 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:28:52.408 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:28:52.408 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:28:52.409 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:28:52.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:28:52.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:28:53.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:28:54.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:28:54.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:28:54.168 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:28:54.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:28:54.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:28:54.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:28:54.813 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:28:55.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:28:55.815 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:28:55.815 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:28:55.815 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:28:55.816 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:28:55.816 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:28:56.289 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:28:57.290 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:28:57.290 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:28:57.291 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:28:57.291 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:28:57.291 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:28:57.291 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:28:57.606 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:28:58.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:28:58.608 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-25T23:28:58.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:28:58.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-25T23:28:58.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:28:58.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:28:58.928 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:28:59.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:28:59.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:28:59.930 INFO:tasks.workunit.client.0.smithi129.stdout:21 2024-03-25T23:28:59.931 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-25T23:28:59.931 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:28:59.931 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:29:00.281 INFO:tasks.workunit.client.0.smithi129.stdout:osd.2 up in weight 1 up_from 34 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/46332891,v1:127.0.0.1:6819/46332891] [v2:127.0.0.1:6820/46332891,v1:127.0.0.1:6821/46332891] exists,up 4c4afab9-e887-4776-85f3-b07114ad18ab 2024-03-25T23:29:00.282 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:29:00.282 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:29:00.282 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:29:00.282 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:283: auto_repair_erasure_coded: create_rbd_pool 2024-03-25T23:29:00.282 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-25T23:29:00.651 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'rbd' does not exist 2024-03-25T23:29:00.669 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-25T23:29:00.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-25T23:29:01.201 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'rbd' created 2024-03-25T23:29:01.223 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T23:29:02.226 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-25T23:29:04.878 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:284: auto_repair_erasure_coded: wait_for_clean 2024-03-25T23:29:04.878 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:29:04.878 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:29:04.878 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:29:04.878 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:29:04.879 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:29:04.879 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:29:04.880 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:29:04.880 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:29:04.880 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:29:05.000 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:29:05.001 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:29:05.001 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:29:05.001 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:29:05.001 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:29:05.001 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:29:05.316 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:29:05.316 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:29:05.317 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-25T23:29:05.317 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:29:05.317 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:29:05.317 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:29:05.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836503 2024-03-25T23:29:05.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836503 2024-03-25T23:29:05.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503' 2024-03-25T23:29:05.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:29:05.485 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:29:05.652 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=68719476754 2024-03-25T23:29:05.652 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 68719476754 2024-03-25T23:29:05.652 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-68719476754' 2024-03-25T23:29:05.652 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:29:05.652 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:29:05.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888073 2024-03-25T23:29:05.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888073 2024-03-25T23:29:05.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-68719476754 2-146028888073' 2024-03-25T23:29:05.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:29:05.797 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836503 2024-03-25T23:29:05.797 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:29:05.798 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:29:05.799 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836503 2024-03-25T23:29:05.799 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:29:05.800 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 21474836503 2024-03-25T23:29:05.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836503 2024-03-25T23:29:05.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-03-25T23:29:05.800 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:29:06.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836503 2024-03-25T23:29:06.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:29:07.143 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:29:07.144 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:29:07.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836503 2024-03-25T23:29:07.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:29:07.462 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-68719476754 2024-03-25T23:29:07.462 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:29:07.463 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:29:07.464 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-68719476754 2024-03-25T23:29:07.464 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:29:07.466 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=68719476754 2024-03-25T23:29:07.466 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 68719476754' 2024-03-25T23:29:07.466 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 68719476754 2024-03-25T23:29:07.466 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:29:07.770 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 68719476754 -lt 68719476754 2024-03-25T23:29:07.771 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:29:07.771 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:29:07.771 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-146028888073 2024-03-25T23:29:07.773 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:29:07.773 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-146028888073 2024-03-25T23:29:07.773 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:29:07.774 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888073 2024-03-25T23:29:07.775 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 146028888073 2024-03-25T23:29:07.775 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 146028888073' 2024-03-25T23:29:07.775 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:29:08.120 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888074 -lt 146028888073 2024-03-25T23:29:08.121 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:29:08.121 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:29:08.121 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:29:08.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-25T23:29:08.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:29:08.494 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:29:08.494 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:29:08.494 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:29:08.494 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:29:08.494 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:29:08.495 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:29:08.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T23:29:08.831 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:29:08.832 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:29:08.832 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:29:09.193 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-25T23:29:09.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:29:09.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:29:09.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: create_ec_pool ecpool false k=2 m=1 2024-03-25T23:29:09.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-25T23:29:09.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-25T23:29:09.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-03-25T23:29:09.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-25T23:29:09.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-03-25T23:29:09.600 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-25T23:29:09.600 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-25T23:29:10.413 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'ecpool' created 2024-03-25T23:29:10.436 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T23:29:11.438 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-03-25T23:29:11.439 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-25T23:29:11.439 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:29:11.439 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:29:11.439 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:29:11.439 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:29:11.439 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:29:11.439 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:29:11.440 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:29:11.440 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:29:11.440 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:29:11.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:29:11.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:29:11.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:29:11.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:29:11.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:29:11.567 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:29:11.882 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:29:11.882 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:29:11.882 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-25T23:29:11.882 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:29:11.882 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:29:11.882 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:29:12.028 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836505 2024-03-25T23:29:12.028 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836505 2024-03-25T23:29:12.028 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505' 2024-03-25T23:29:12.028 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:29:12.028 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:29:12.216 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=68719476756 2024-03-25T23:29:12.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 68719476756 2024-03-25T23:29:12.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-68719476756' 2024-03-25T23:29:12.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:29:12.217 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:29:12.369 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888076 2024-03-25T23:29:12.369 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888076 2024-03-25T23:29:12.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-68719476756 2-146028888076' 2024-03-25T23:29:12.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:29:12.370 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836505 2024-03-25T23:29:12.370 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:29:12.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:29:12.372 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836505 2024-03-25T23:29:12.372 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:29:12.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836505 2024-03-25T23:29:12.373 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 21474836505 2024-03-25T23:29:12.374 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-03-25T23:29:12.374 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:29:12.685 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836505 2024-03-25T23:29:12.686 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:29:13.687 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:29:13.688 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:29:14.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836505 2024-03-25T23:29:14.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:29:14.010 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-68719476756 2024-03-25T23:29:14.011 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:29:14.014 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:29:14.014 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-68719476756 2024-03-25T23:29:14.014 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:29:14.015 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 68719476756 2024-03-25T23:29:14.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=68719476756 2024-03-25T23:29:14.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 68719476756' 2024-03-25T23:29:14.015 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:29:14.338 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 68719476756 -lt 68719476756 2024-03-25T23:29:14.338 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:29:14.339 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-146028888076 2024-03-25T23:29:14.339 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:29:14.340 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:29:14.341 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:29:14.341 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-146028888076 2024-03-25T23:29:14.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888076 2024-03-25T23:29:14.343 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 146028888076 2024-03-25T23:29:14.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 146028888076' 2024-03-25T23:29:14.343 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:29:14.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888076 -lt 146028888076 2024-03-25T23:29:14.660 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:29:14.661 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:29:14.661 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:29:15.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:29:15.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:29:15.043 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:29:15.043 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:29:15.044 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:29:15.044 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:29:15.044 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:29:15.044 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:29:15.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:29:15.414 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:29:15.415 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:29:15.415 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:29:15.793 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:29:15.793 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:29:15.793 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:29:15.793 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-25T23:29:15.793 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:290: auto_repair_erasure_coded: local payload=ABCDEF 2024-03-25T23:29:15.793 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:291: auto_repair_erasure_coded: echo ABCDEF 2024-03-25T23:29:15.793 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:292: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-25T23:29:16.041 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-03-25T23:29:16.041 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-03-25T23:29:16.042 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-25T23:29:16.042 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-03-25T23:29:16.042 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-03-25T23:29:16.043 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-25T23:29:16.043 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-03-25T23:29:16.043 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-25T23:29:16.386 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-03-25T23:29:16.387 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-03-25T23:29:16.387 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-03-25T23:29:16.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-25T23:29:16.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:29:16.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:29:16.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-25T23:29:16.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:29:16.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-25T23:29:16.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:29:16.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:29:16.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-25T23:29:16.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:29:16.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-25T23:29:16.712 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:29:16.712 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:29:16.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:29:16.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:29:16.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:29:18.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:29:18.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-25T23:29:18.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:29:18.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:29:18.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-25T23:29:18.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:29:18.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:29:18.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-25T23:29:19.472 INFO:tasks.workunit.client.0.smithi129.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-03-25T23:29:19.473 INFO:tasks.workunit.client.0.smithi129.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-03-25T23:29:20.014 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:29:20.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:29:20.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:29:20.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-25T23:29:20.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:29:20.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:29:20.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:29:20.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:29:20.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:29:20.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:29:20.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-25T23:29:20.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-25T23:29:20.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:29:20.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:29:20.016 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:29:20.016 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:29:20.016 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:29:20.016 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:29:20.016 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:29:20.016 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:29:20.016 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:29:20.017 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:29:20.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:29:20.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:29:20.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:29:20.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:29:20.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:29:20.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:29:20.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:29:20.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:29:20.018 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:29:20.018 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-25T23:29:20.018 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-25T23:29:20.018 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-25T23:29:20.018 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.1 2024-03-25T23:29:20.018 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:29:20.019 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-25T23:29:20.019 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-25T23:29:20.019 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:29:20.020 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:29:20.020 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:29:20.055 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:29:20.056+0000 7fec4ad127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:29:20.058 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:29:20.056+0000 7fec4ad127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:29:20.059 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:29:20.060+0000 7fec4ad127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:29:20.333 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-25T23:29:20.333 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:29:20.333 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T23:29:20.333 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:29:20.333 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:29:20.333 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:29:20.333 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:29:20.333 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-25T23:29:20.333 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:29:20.334 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:29:20.600 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:29:20.600+0000 7fec4ad127c0 -1 Falling back to public interface 2024-03-25T23:29:20.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:29:21.655 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:29:21.655 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:29:21.655 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:29:21.655 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:29:21.655 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:29:21.655 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:29:22.029 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:29:23.031 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:29:23.031 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:29:23.031 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:29:23.031 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:29:23.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:29:23.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:29:23.477 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:29:23.697 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:29:23.696+0000 7fec4ad127c0 -1 osd.1 53 log_to_monitors true 2024-03-25T23:29:24.479 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:29:24.479 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:29:24.479 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:29:24.479 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:29:24.480 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:29:24.480 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:29:24.867 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1 up in weight 1 up_from 61 up_thru 61 down_at 55 last_clean_interval [16,53) [v2:127.0.0.1:6810/3027808357,v1:127.0.0.1:6811/3027808357] [v2:127.0.0.1:6812/3027808357,v1:127.0.0.1:6813/3027808357] exists,up 1925eaee-ecba-41a8-9e4f-46b8035d3274 2024-03-25T23:29:24.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:29:24.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:29:24.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:29:24.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:29:24.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:29:24.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:29:24.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:29:24.868 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:29:24.868 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:29:24.868 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:29:24.868 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:29:24.868 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:29:24.869 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:29:24.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:29:24.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:29:24.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:29:24.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:29:24.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:29:24.967 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:29:25.308 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:29:25.308 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:29:25.308 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-25T23:29:25.308 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:29:25.308 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:29:25.309 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:29:25.555 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836509 2024-03-25T23:29:25.555 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836509 2024-03-25T23:29:25.555 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509' 2024-03-25T23:29:25.555 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:29:25.555 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:29:25.695 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=261993005058 2024-03-25T23:29:25.695 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 261993005058 2024-03-25T23:29:25.695 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-261993005058' 2024-03-25T23:29:25.695 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:29:25.695 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:29:25.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888079 2024-03-25T23:29:25.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888079 2024-03-25T23:29:25.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-261993005058 2-146028888079' 2024-03-25T23:29:25.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:29:25.919 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836509 2024-03-25T23:29:25.919 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:29:25.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:29:25.919 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836509 2024-03-25T23:29:25.920 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:29:25.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836509 2024-03-25T23:29:25.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-03-25T23:29:25.920 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 21474836509 2024-03-25T23:29:25.920 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:29:26.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836509 2024-03-25T23:29:26.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:29:27.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:29:27.285 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:29:27.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836509 -lt 21474836509 2024-03-25T23:29:27.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:29:27.608 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-261993005058 2024-03-25T23:29:27.608 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:29:27.610 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:29:27.611 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-261993005058 2024-03-25T23:29:27.611 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:29:27.612 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=261993005058 2024-03-25T23:29:27.612 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 261993005058' 2024-03-25T23:29:27.612 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 261993005058 2024-03-25T23:29:27.613 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:29:28.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 261993005058 -lt 261993005058 2024-03-25T23:29:28.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:29:28.168 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-146028888079 2024-03-25T23:29:28.169 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:29:28.171 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:29:28.171 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-146028888079 2024-03-25T23:29:28.171 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:29:28.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888079 2024-03-25T23:29:28.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 146028888079' 2024-03-25T23:29:28.173 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 146028888079 2024-03-25T23:29:28.173 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:29:28.503 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888080 -lt 146028888079 2024-03-25T23:29:28.504 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:29:28.505 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:29:28.505 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:29:28.893 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:29:28.893 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:29:28.894 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:29:28.894 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:29:28.895 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:29:28.895 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:29:28.895 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:29:28.895 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:29:29.235 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:29:29.236 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:29:29.236 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:29:29.236 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:29:29.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:29:29.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:29:29.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:29:29.599 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2024-03-25T23:29:29.599 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-25T23:29:29.599 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-25T23:29:29.599 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-25T23:29:29.599 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-25T23:29:29.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: local pgid=2.0 2024-03-25T23:29:29.915 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2024-03-25T23:29:29.915 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:29:29.915 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:29:29.916 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:29:29.916 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:29:30.333 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: wait_for_scrub 2.0 2024-03-25T23:29:15.506013+0000 2024-03-25T23:29:30.333 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-25T23:29:30.333 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-25T23:29:15.506013+0000 2024-03-25T23:29:30.333 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-25T23:29:30.333 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-25T23:29:30.334 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:29:30.334 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:29:30.334 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:29:30.334 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:29:30.334 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:29:30.334 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:29:30.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:29:15.506013+0000 '>' 2024-03-25T23:29:15.506013+0000 2024-03-25T23:29:30.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:29:31.625 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:29:31.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:29:31.626 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:29:31.626 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:29:31.627 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:29:31.627 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:29:31.627 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:29:31.950 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:29:15.506013+0000 '>' 2024-03-25T23:29:15.506013+0000 2024-03-25T23:29:31.951 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:29:32.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:29:32.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:29:32.953 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:29:32.953 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:29:32.953 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:29:32.953 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:29:32.953 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:29:33.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:29:29.415076+0000 '>' 2024-03-25T23:29:15.506013+0000 2024-03-25T23:29:33.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-25T23:29:33.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:300: auto_repair_erasure_coded: wait_for_clean 2024-03-25T23:29:33.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:29:33.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:29:33.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:29:33.309 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:29:33.309 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:29:33.310 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:29:33.310 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:29:33.310 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:29:33.310 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:29:33.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:29:33.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:29:33.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:29:33.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:29:33.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:29:33.420 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:29:33.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:29:33.777 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:29:33.777 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-25T23:29:33.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:29:33.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:29:33.778 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:29:33.928 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836512 2024-03-25T23:29:33.928 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836512 2024-03-25T23:29:33.928 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512' 2024-03-25T23:29:33.928 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:29:33.928 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:29:34.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=261993005061 2024-03-25T23:29:34.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 261993005061 2024-03-25T23:29:34.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512 1-261993005061' 2024-03-25T23:29:34.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:29:34.093 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:29:34.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888082 2024-03-25T23:29:34.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888082 2024-03-25T23:29:34.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512 1-261993005061 2-146028888082' 2024-03-25T23:29:34.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:29:34.237 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836512 2024-03-25T23:29:34.237 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:29:34.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:29:34.239 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836512 2024-03-25T23:29:34.239 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:29:34.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836512 2024-03-25T23:29:34.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-03-25T23:29:34.240 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 21474836512 2024-03-25T23:29:34.240 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:29:34.551 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836511 -lt 21474836512 2024-03-25T23:29:34.552 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:29:35.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:29:35.554 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:29:35.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836512 -lt 21474836512 2024-03-25T23:29:35.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:29:35.884 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-261993005061 2024-03-25T23:29:35.884 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:29:35.886 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:29:35.887 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-261993005061 2024-03-25T23:29:35.887 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:29:35.888 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=261993005061 2024-03-25T23:29:35.888 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 261993005061 2024-03-25T23:29:35.889 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 261993005061' 2024-03-25T23:29:35.889 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:29:36.204 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 261993005061 -lt 261993005061 2024-03-25T23:29:36.204 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:29:36.205 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-146028888082 2024-03-25T23:29:36.205 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:29:36.206 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:29:36.207 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-146028888082 2024-03-25T23:29:36.207 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:29:36.208 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888082 2024-03-25T23:29:36.208 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 146028888082' 2024-03-25T23:29:36.209 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 146028888082 2024-03-25T23:29:36.209 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:29:36.531 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888082 -lt 146028888082 2024-03-25T23:29:36.532 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:29:36.533 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:29:36.533 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:29:37.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:29:37.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:29:37.037 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:29:37.038 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:29:37.038 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:29:37.038 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:29:37.038 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:29:37.038 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:29:37.411 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:29:37.411 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:29:37.411 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:29:37.412 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:29:37.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:29:37.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:29:37.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:29:37.780 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-03-25T23:29:37.780 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-03-25T23:29:37.780 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-25T23:29:37.781 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-03-25T23:29:37.781 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-03-25T23:29:37.781 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-25T23:29:37.781 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-03-25T23:29:37.781 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-25T23:29:38.131 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-03-25T23:29:38.131 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-03-25T23:29:38.131 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-03-25T23:29:38.446 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-25T23:29:38.446 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:29:38.446 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:29:38.446 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-25T23:29:38.446 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:29:38.447 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-25T23:29:38.447 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:29:38.447 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:29:38.447 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-25T23:29:38.447 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:29:38.447 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-25T23:29:38.447 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:29:38.447 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:29:38.447 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:29:38.447 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:29:38.447 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:29:39.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:29:39.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-25T23:29:39.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:29:39.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:29:39.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-25T23:29:39.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:29:39.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:29:39.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-25T23:29:40.050 INFO:tasks.workunit.client.0.smithi129.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-03-25T23:29:40.051 INFO:tasks.workunit.client.0.smithi129.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-03-25T23:29:40.051 INFO:tasks.workunit.client.0.smithi129.stdout:_ 2024-03-25T23:29:40.051 INFO:tasks.workunit.client.0.smithi129.stdout:hinfo_key 2024-03-25T23:29:40.051 INFO:tasks.workunit.client.0.smithi129.stdout:snapset 2024-03-25T23:29:40.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:29:40.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:29:40.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:29:40.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-25T23:29:40.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:29:40.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:29:40.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:29:40.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:29:40.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:29:40.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:29:40.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-25T23:29:40.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-25T23:29:40.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:29:40.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:29:40.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:29:40.348 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:29:40.348 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:29:40.348 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:29:40.348 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:29:40.348 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:29:40.348 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:29:40.350 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:29:40.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:29:40.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:29:40.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:29:40.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:29:40.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:29:40.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:29:40.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:29:40.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:29:40.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:29:40.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-25T23:29:40.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-25T23:29:40.351 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-25T23:29:40.351 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.1 2024-03-25T23:29:40.351 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:29:40.351 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-25T23:29:40.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-25T23:29:40.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:29:40.353 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:29:40.353 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:29:40.395 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:29:40.395+0000 7fa6f1fe07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:29:40.397 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:29:40.395+0000 7fa6f1fe07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:29:40.399 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:29:40.399+0000 7fa6f1fe07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:29:40.652 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-25T23:29:40.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:29:40.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T23:29:40.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:29:40.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:29:40.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:29:40.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:29:40.653 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-25T23:29:40.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:29:40.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:29:40.933 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:29:40.931+0000 7fa6f1fe07c0 -1 Falling back to public interface 2024-03-25T23:29:40.985 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:29:41.986 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:29:41.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:29:41.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:29:41.987 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:29:41.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:29:41.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:29:42.312 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:29:43.313 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:29:43.313 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:29:43.314 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:29:43.314 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:29:43.314 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:29:43.314 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:29:43.708 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:29:44.140 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:29:44.139+0000 7fa6f1fe07c0 -1 osd.1 71 log_to_monitors true 2024-03-25T23:29:44.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:29:44.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:29:44.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:29:44.711 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:29:44.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:29:44.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:29:45.183 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1 up in weight 1 up_from 77 up_thru 61 down_at 72 last_clean_interval [61,71) [v2:127.0.0.1:6810/4105473282,v1:127.0.0.1:6811/4105473282] [v2:127.0.0.1:6812/4105473282,v1:127.0.0.1:6813/4105473282] exists,up 1925eaee-ecba-41a8-9e4f-46b8035d3274 2024-03-25T23:29:45.183 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:29:45.183 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:29:45.183 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:29:45.183 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:29:45.183 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:29:45.183 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:29:45.183 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:29:45.184 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:29:45.185 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:29:45.185 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:29:45.185 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:29:45.185 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:29:45.185 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:29:45.296 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:29:45.296 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:29:45.296 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:29:45.296 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:29:45.296 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:29:45.297 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:29:45.717 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:29:45.717 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:29:45.717 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-25T23:29:45.717 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:29:45.718 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:29:45.718 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:29:45.978 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836515 2024-03-25T23:29:45.978 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836515 2024-03-25T23:29:45.978 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836515' 2024-03-25T23:29:45.978 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:29:45.979 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:29:46.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481794 2024-03-25T23:29:46.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481794 2024-03-25T23:29:46.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836515 1-330712481794' 2024-03-25T23:29:46.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:29:46.166 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:29:46.314 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888086 2024-03-25T23:29:46.315 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888086 2024-03-25T23:29:46.315 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836515 1-330712481794 2-146028888086' 2024-03-25T23:29:46.315 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:29:46.315 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836515 2024-03-25T23:29:46.315 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:29:46.316 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:29:46.317 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836515 2024-03-25T23:29:46.317 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:29:46.318 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836515 2024-03-25T23:29:46.318 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836515' 2024-03-25T23:29:46.318 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 21474836515 2024-03-25T23:29:46.318 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:29:46.656 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836514 -lt 21474836515 2024-03-25T23:29:46.656 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:29:47.657 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:29:47.658 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:29:47.981 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836515 -lt 21474836515 2024-03-25T23:29:47.981 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:29:47.982 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:29:47.982 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-330712481794 2024-03-25T23:29:47.983 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:29:47.984 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-330712481794 2024-03-25T23:29:47.984 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:29:47.985 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481794 2024-03-25T23:29:47.985 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 330712481794' 2024-03-25T23:29:47.985 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 330712481794 2024-03-25T23:29:47.986 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:29:48.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481794 -lt 330712481794 2024-03-25T23:29:48.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:29:48.341 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-146028888086 2024-03-25T23:29:48.341 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:29:48.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:29:48.344 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-146028888086 2024-03-25T23:29:48.344 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:29:48.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888086 2024-03-25T23:29:48.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 146028888086' 2024-03-25T23:29:48.346 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 146028888086 2024-03-25T23:29:48.346 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:29:48.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888086 -lt 146028888086 2024-03-25T23:29:48.655 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:29:48.655 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:29:48.655 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:29:49.007 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:29:49.007 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:29:49.008 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:29:49.008 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:29:49.008 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:29:49.008 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:29:49.008 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:29:49.008 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:29:49.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-25T23:29:49.352 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:29:49.353 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:29:49.353 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:29:49.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-25T23:29:49.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-25T23:29:49.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-25T23:29:49.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-25T23:29:49.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:29:49.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-25T23:29:49.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:29:49.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:29:49.833 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:29:49.833 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:29:49.833 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:29:49.833 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:29:49.834 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:29:49.834 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:29:50.133 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-25T23:29:50.133 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:29:50.133 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:29:50.134 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:29:50.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-25T23:29:50.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-25T23:29:50.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-25T23:29:50.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-25T23:29:50.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-25T23:29:50.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-25T23:29:50.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-25T23:29:50.588 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-25T23:29:50.588 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-25T23:29:50.970 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-25T23:29:50.970 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-25T23:29:50.970 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-25T23:29:50.970 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:29:50.970 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-25T23:29:51.171 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:29:51.171 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:29:51.172 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:29:51.172 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:29:51.172 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:29:51.172 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:29:51.172 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:29:51.173 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:29:51.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-25T23:29:51.484 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:29:51.485 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:29:51.485 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:29:51.847 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-25T23:29:51.848 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-25T23:29:51.848 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-25T23:29:51.848 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-25T23:29:51.848 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-25T23:29:51.848 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-25T23:29:51.848 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-25T23:29:51.849 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-25T23:29:51.849 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-25T23:29:52.250 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-25T23:29:52.250 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-25T23:29:52.250 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-25T23:29:52.250 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:29:52.250 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-25T23:29:52.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:29:52.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:29:52.652 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:29:52.652 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:29:52.652 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:29:52.652 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:29:52.653 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:29:52.653 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:29:52.974 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:29:52.975 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:29:52.975 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:29:52.975 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:29:53.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:29:53.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:29:53.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:29:53.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-25T23:29:53.537 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:305: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-25T23:29:53.540 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-25T23:29:53.540 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-25T23:29:53.540 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T23:29:53.540 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-25T23:29:53.540 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:29:53.540 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:29:53.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:29:53.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:29:53.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:29:53.853 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:29:53.854 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T23:29:53.855 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T23:29:53.855 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T23:29:53.858 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T23:29:53.858 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T23:29:53.858 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T23:29:53.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:29:53.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T23:29:53.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T23:29:53.861 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:29:53.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T23:29:53.864 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T23:29:53.864 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-25T23:29:53.889 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T23:29:53.889 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:29:53.889 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:29:53.889 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-25T23:29:53.891 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T23:29:53.891 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T23:29:53.891 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-25T23:29:53.891 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-25T23:29:53.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-25T23:29:53.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-25T23:29:53.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-25T23:29:53.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T23:29:53.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-25T23:29:53.892 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:29:53.892 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:29:53.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:29:53.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:29:53.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:29:53.894 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:29:53.894 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T23:29:53.895 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T23:29:53.896 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T23:29:53.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T23:29:53.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T23:29:53.898 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T23:29:53.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:29:53.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T23:29:53.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T23:29:53.899 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:29:53.900 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T23:29:53.902 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T23:29:53.902 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-25T23:29:53.903 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T23:29:53.904 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:29:53.904 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:29:53.904 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-25T23:29:53.905 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T23:29:53.905 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T23:29:53.905 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-25T23:29:53.906 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-25T23:29:53.906 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:29:53.907 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:29:53.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61636 2024-03-25T23:29:53.908 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-25T23:29:53.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-25T23:29:53.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-25T23:29:53.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-25T23:29:53.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_erasure_coded_overwrites td/osd-scrub-repair 2024-03-25T23:29:53.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:313: TEST_auto_repair_erasure_coded_overwrites: '[' true = true ']' 2024-03-25T23:29:53.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: TEST_auto_repair_erasure_coded_overwrites: auto_repair_erasure_coded td/osd-scrub-repair true 2024-03-25T23:29:53.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-03-25T23:29:53.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: auto_repair_erasure_coded: local allow_overwrites=true 2024-03-25T23:29:53.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: auto_repair_erasure_coded: local poolname=ecpool 2024-03-25T23:29:53.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2024-03-25T23:29:53.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-25T23:29:53.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-25T23:29:53.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-25T23:29:53.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-25T23:29:53.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-25T23:29:53.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-25T23:29:54.402 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-25T23:29:54.402 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:29:54.402 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:29:54.402 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:29:54.402 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:29:54.403 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:29:54.403 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:29:54.403 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-25T23:29:54.622 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-25T23:29:54.623 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-25T23:29:54.623 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T23:29:54.624 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T23:29:54.624 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-25T23:29:54.624 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-25T23:29:54.624 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T23:29:54.624 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T23:29:54.624 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T23:29:54.625 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T23:29:54.625 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:29:54.625 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:29:54.625 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-25T23:29:54.626 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T23:29:54.626 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get fsid 2024-03-25T23:29:54.745 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-25T23:29:54.746 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T23:29:54.746 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T23:29:54.746 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-25T23:29:54.746 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T23:29:54.746 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-25T23:29:54.746 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T23:29:54.747 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T23:29:54.747 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T23:29:54.747 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:29:54.748 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:29:54.748 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-25T23:29:54.748 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T23:29:54.748 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get mon_host 2024-03-25T23:29:54.853 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:274: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2024-03-25T23:29:54.853 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-25T23:29:54.853 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-25T23:29:54.853 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-25T23:29:54.853 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-25T23:29:54.853 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-25T23:29:54.853 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-25T23:29:55.704 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-25T23:29:55.704 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:29:55.704 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:29:55.704 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:29:55.704 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:29:55.704 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:29:55.704 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:29:55.705 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T23:29:55.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T23:29:55.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-25T23:29:55.740 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: seq 0 2 2024-03-25T23:29:55.744 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-03-25T23:29:55.744 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:29:55.744 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:29:55.744 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:29:55.744 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-25T23:29:55.744 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:29:55.744 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-25T23:29:55.744 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:29:55.744 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:29:55.744 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:29:55.744 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:29:55.744 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-25T23:29:55.745 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-25T23:29:55.745 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:29:55.745 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:29:55.745 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:29:55.745 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:29:55.745 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:29:55.745 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:29:55.745 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:29:55.745 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:29:55.745 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:29:55.745 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:29:55.745 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:29:55.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:29:55.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:29:55.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:29:55.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:29:55.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:29:55.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:29:55.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:29:55.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:29:55.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:29:55.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:29:55.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-25T23:29:55.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-25T23:29:55.747 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:29:55.747 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f8a9bb9a-77c1-44c3-b785-180d13d17757 2024-03-25T23:29:55.747 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f8a9bb9a-77c1-44c3-b785-180d13d17757' 2024-03-25T23:29:55.747 INFO:tasks.workunit.client.0.smithi129.stdout:add osd0 f8a9bb9a-77c1-44c3-b785-180d13d17757 2024-03-25T23:29:55.748 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:29:55.765 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDzCAJm9/vALRAAx8kZwKC8vZxfcPkNM1URIA== 2024-03-25T23:29:55.765 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDzCAJm9/vALRAAx8kZwKC8vZxfcPkNM1URIA=="}' 2024-03-25T23:29:55.766 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f8a9bb9a-77c1-44c3-b785-180d13d17757 -i td/osd-scrub-repair/0/new.json 2024-03-25T23:29:55.999 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:29:56.014 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-25T23:29:56.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDzCAJm9/vALRAAx8kZwKC8vZxfcPkNM1URIA== --osd-uuid f8a9bb9a-77c1-44c3-b785-180d13d17757 2024-03-25T23:29:56.048 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:29:56.047+0000 7fb78175a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:29:56.050 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:29:56.051+0000 7fb78175a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:29:56.051 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:29:56.051+0000 7fb78175a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:29:56.052 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:29:56.051+0000 7fb78175a7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-25T23:29:56.052 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:29:56.051+0000 7fb78175a7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-25T23:29:56.052 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:29:56.051+0000 7fb78175a7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-25T23:29:56.150 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:29:56.147+0000 7fb78175a7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-25T23:30:02.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-25T23:30:02.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:30:02.389 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd0 key to auth repository 2024-03-25T23:30:02.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-25T23:30:02.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:30:02.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-25T23:30:02.784 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.0 2024-03-25T23:30:02.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:30:02.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:30:02.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:30:02.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:30:02.821 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:30:02.819+0000 7ff7e9bd17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:30:02.848 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:30:02.847+0000 7ff7e9bd17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:30:02.849 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:30:02.847+0000 7ff7e9bd17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:30:03.118 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-25T23:30:03.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:30:03.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T23:30:03.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:30:03.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:30:03.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:30:03.119 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:30:03.120 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:30:03.120 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:30:03.120 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:30:03.400 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:30:03.399+0000 7ff7e9bd17c0 -1 Falling back to public interface 2024-03-25T23:30:03.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:30:04.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:30:04.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:30:04.460 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:30:04.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:30:04.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:30:04.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:30:04.764 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:30:05.765 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:30:05.766 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:30:05.766 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:30:05.766 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:30:05.766 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:30:05.766 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:30:05.982 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:30:05.979+0000 7ff7e9bd17c0 -1 osd.0 0 log_to_monitors true 2024-03-25T23:30:06.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:30:07.104 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:30:07.104 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:30:07.104 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:30:07.105 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:30:07.105 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:30:07.105 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:30:07.425 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:30:08.426 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:30:08.427 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:30:08.427 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:30:08.427 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:30:08.427 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:30:08.427 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:30:08.808 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:30:09.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:30:09.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:30:09.811 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:30:09.811 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:30:09.811 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:30:09.811 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:30:10.121 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:30:11.122 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:30:11.123 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:30:11.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:30:11.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:30:11.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:30:11.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:30:11.441 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:30:12.443 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:30:12.443 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:30:12.443 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:30:12.444 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:30:12.444 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:30:12.444 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:30:12.851 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:30:13.853 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:30:13.854 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:30:13.854 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:30:13.854 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:30:13.854 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:30:13.855 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:30:14.159 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:30:15.161 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:30:15.161 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:30:15.161 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:30:15.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:30:15.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:30:15.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:30:15.475 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:30:16.477 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:30:16.477 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:30:16.477 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:30:16.478 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:30:16.478 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:30:16.478 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:30:16.852 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:30:17.853 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:30:17.854 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:30:17.854 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:30:17.854 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:30:17.854 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:30:17.854 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:30:18.171 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:30:19.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:30:19.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:30:19.174 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:30:19.174 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:30:19.174 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:30:19.174 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:30:19.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:30:20.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:30:20.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:30:20.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:30:20.497 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:30:20.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:30:20.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:30:20.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:30:21.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:30:21.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:30:21.871 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:30:21.871 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:30:21.871 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:30:21.871 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:30:22.340 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:30:23.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:30:23.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:30:23.342 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:30:23.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:30:23.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:30:23.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:30:23.866 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:30:24.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:30:24.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:30:24.868 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:30:24.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:30:24.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:30:24.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:30:25.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:30:26.233 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:30:26.233 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:30:26.234 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:30:26.234 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:30:26.234 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:30:26.234 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:30:26.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:30:27.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:30:27.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:30:27.731 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:30:27.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:30:27.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:30:27.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:30:28.566 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:30:29.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:30:29.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:30:29.569 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:30:29.569 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:30:29.569 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:30:29.569 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:30:30.131 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:30:31.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:30:31.133 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:30:31.133 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-25T23:30:31.133 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-25T23:30:31.133 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:30:31.133 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:30:31.448 INFO:tasks.workunit.client.0.smithi129.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2044832871,v1:127.0.0.1:6803/2044832871] [v2:127.0.0.1:6804/2044832871,v1:127.0.0.1:6805/2044832871] exists,up f8a9bb9a-77c1-44c3-b785-180d13d17757 2024-03-25T23:30:31.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:30:31.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:30:31.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:30:31.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-03-25T23:30:31.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:30:31.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:30:31.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:30:31.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-25T23:30:31.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:30:31.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:30:31.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:30:31.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:30:31.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:30:31.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:30:31.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-25T23:30:31.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-25T23:30:31.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:30:31.454 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:30:31.454 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:30:31.454 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:30:31.454 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:30:31.454 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:30:31.454 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:30:31.454 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:30:31.454 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:30:31.454 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:30:31.454 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:30:31.454 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:30:31.454 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:30:31.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:30:31.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:30:31.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:30:31.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:30:31.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:30:31.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:30:31.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:30:31.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:30:31.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-25T23:30:31.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-25T23:30:31.458 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:30:31.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e17236c7-d76e-42d7-ba66-309769fab2f8 2024-03-25T23:30:31.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 e17236c7-d76e-42d7-ba66-309769fab2f8' 2024-03-25T23:30:31.459 INFO:tasks.workunit.client.0.smithi129.stdout:add osd1 e17236c7-d76e-42d7-ba66-309769fab2f8 2024-03-25T23:30:31.459 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:30:31.478 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAXCQJmzradHBAApDb9zrlJa8ic7Tz1YPyoWA== 2024-03-25T23:30:31.478 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAXCQJmzradHBAApDb9zrlJa8ic7Tz1YPyoWA=="}' 2024-03-25T23:30:31.478 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e17236c7-d76e-42d7-ba66-309769fab2f8 -i td/osd-scrub-repair/1/new.json 2024-03-25T23:30:31.862 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:30:31.880 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-25T23:30:31.882 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAXCQJmzradHBAApDb9zrlJa8ic7Tz1YPyoWA== --osd-uuid e17236c7-d76e-42d7-ba66-309769fab2f8 2024-03-25T23:30:31.920 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:30:31.918+0000 7f870dd0b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:30:31.964 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:30:31.962+0000 7f870dd0b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:30:31.966 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:30:31.962+0000 7f870dd0b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:30:31.966 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:30:31.962+0000 7f870dd0b7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-25T23:30:31.966 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:30:31.962+0000 7f870dd0b7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-25T23:30:31.966 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:30:31.962+0000 7f870dd0b7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-25T23:30:32.043 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:30:32.042+0000 7f870dd0b7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-25T23:30:38.984 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-25T23:30:38.984 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:30:38.985 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd1 key to auth repository 2024-03-25T23:30:38.985 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-25T23:30:38.986 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:30:39.382 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.1 2024-03-25T23:30:39.383 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-25T23:30:39.383 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:30:39.383 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:30:39.383 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:30:39.384 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:30:39.418 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:30:39.414+0000 7fe93993b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:30:39.446 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:30:39.442+0000 7fe93993b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:30:39.447 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:30:39.446+0000 7fe93993b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:30:39.693 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-25T23:30:39.693 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:30:39.693 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T23:30:39.693 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:30:39.693 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:30:39.693 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:30:39.693 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:30:39.694 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:30:39.694 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:30:39.694 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:30:39.996 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:30:39.994+0000 7fe93993b7c0 -1 Falling back to public interface 2024-03-25T23:30:40.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:30:41.011 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:30:41.011 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:30:41.012 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:30:41.012 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:30:41.012 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:30:41.012 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:30:41.356 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:30:42.358 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:30:42.358 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:30:42.358 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:30:42.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:30:42.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:30:42.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:30:42.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:30:42.810 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:30:42.806+0000 7fe93993b7c0 -1 osd.1 0 log_to_monitors true 2024-03-25T23:30:43.678 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:30:43.678 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:30:43.679 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:30:43.679 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:30:43.679 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:30:43.680 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:30:44.143 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:30:45.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:30:45.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:30:45.146 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:30:45.146 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:30:45.146 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:30:45.146 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:30:45.529 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:30:46.531 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:30:46.531 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:30:46.532 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:30:46.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:30:46.533 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:30:46.533 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:30:47.035 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:30:48.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:30:48.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:30:48.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:30:48.037 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:30:48.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:30:48.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:30:48.362 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:30:49.364 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:30:49.364 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:30:49.364 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:30:49.364 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:30:49.365 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:30:49.365 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:30:49.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:30:50.759 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:30:50.759 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:30:50.760 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:30:50.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:30:50.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:30:50.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:30:51.133 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:30:52.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:30:52.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:30:52.135 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:30:52.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:30:52.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:30:52.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:30:52.639 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:30:53.641 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:30:53.641 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:30:53.641 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:30:53.642 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:30:53.642 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:30:53.642 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:30:54.108 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:30:55.110 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:30:55.110 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:30:55.111 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:30:55.111 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:30:55.111 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:30:55.111 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:30:55.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:30:56.549 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:30:56.549 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:30:56.549 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:30:56.549 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:30:56.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:30:56.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:30:56.913 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:30:57.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:30:57.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:30:57.915 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:30:57.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:30:57.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:30:57.916 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:30:58.385 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:30:59.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:30:59.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:30:59.387 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:30:59.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:30:59.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:30:59.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:31:00.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:31:01.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:31:01.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:31:01.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:31:01.460 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:31:01.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:31:01.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:31:02.308 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:31:03.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:31:03.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:31:03.310 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:31:03.311 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:31:03.311 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:31:03.311 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:31:03.684 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:31:04.686 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:31:04.686 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:31:04.687 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:31:04.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:31:04.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:31:04.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:31:05.449 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:31:06.451 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:31:06.451 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:31:06.451 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:31:06.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:31:06.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:31:06.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:31:07.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:31:08.203 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:31:08.203 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:31:08.203 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:31:08.204 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:31:08.204 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:31:08.204 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:31:08.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:31:09.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:31:09.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:31:09.662 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-25T23:31:09.662 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-25T23:31:09.662 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:31:09.662 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:31:09.982 INFO:tasks.workunit.client.0.smithi129.stdout:osd.1 up in weight 1 up_from 16 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1570182214,v1:127.0.0.1:6811/1570182214] [v2:127.0.0.1:6812/1570182214,v1:127.0.0.1:6813/1570182214] exists,up e17236c7-d76e-42d7-ba66-309769fab2f8 2024-03-25T23:31:09.986 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:31:09.986 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:31:09.986 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:31:09.986 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-03-25T23:31:09.986 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:31:09.986 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:31:09.986 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:31:09.986 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-25T23:31:09.986 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:31:09.986 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-25T23:31:09.986 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:31:09.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:31:09.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:31:09.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:31:09.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-25T23:31:09.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-25T23:31:09.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:31:09.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:31:09.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:31:09.987 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:31:09.987 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:31:09.987 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:31:09.988 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:31:09.988 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:31:09.988 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:31:09.988 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:31:09.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:31:09.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:31:09.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:31:09.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:31:09.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:31:09.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:31:09.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:31:09.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:31:09.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:31:09.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:31:09.989 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:31:09.989 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-25T23:31:09.989 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-25T23:31:09.989 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:31:09.989 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fb6d380b-0822-4d14-bb7d-c99654850a39 2024-03-25T23:31:09.990 INFO:tasks.workunit.client.0.smithi129.stdout:add osd2 fb6d380b-0822-4d14-bb7d-c99654850a39 2024-03-25T23:31:09.990 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 fb6d380b-0822-4d14-bb7d-c99654850a39' 2024-03-25T23:31:09.990 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:31:10.009 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA+CQJmdiu5ABAAP8wQi5JK4OxYfZPNLOrdtQ== 2024-03-25T23:31:10.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA+CQJmdiu5ABAAP8wQi5JK4OxYfZPNLOrdtQ=="}' 2024-03-25T23:31:10.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fb6d380b-0822-4d14-bb7d-c99654850a39 -i td/osd-scrub-repair/2/new.json 2024-03-25T23:31:10.566 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:31:10.589 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-25T23:31:10.590 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQA+CQJmdiu5ABAAP8wQi5JK4OxYfZPNLOrdtQ== --osd-uuid fb6d380b-0822-4d14-bb7d-c99654850a39 2024-03-25T23:31:10.629 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:31:10.630+0000 7f5f0ab0b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:31:10.668 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:31:10.670+0000 7f5f0ab0b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:31:10.669 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:31:10.670+0000 7f5f0ab0b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:31:10.670 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:31:10.670+0000 7f5f0ab0b7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-25T23:31:10.670 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:31:10.670+0000 7f5f0ab0b7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-25T23:31:10.670 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:31:10.670+0000 7f5f0ab0b7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-25T23:31:10.746 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:31:10.746+0000 7f5f0ab0b7c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-25T23:31:17.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-25T23:31:17.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:31:17.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-25T23:31:17.561 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd2 key to auth repository 2024-03-25T23:31:17.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:31:17.956 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.2 2024-03-25T23:31:17.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-25T23:31:17.957 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:31:17.957 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:31:17.957 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:31:17.957 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:31:17.988 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:31:17.989+0000 7f726eb327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:31:18.020 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:31:18.021+0000 7f726eb327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:31:18.021 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:31:18.021+0000 7f726eb327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:31:18.291 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-25T23:31:18.291 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:31:18.291 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-25T23:31:18.291 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:31:18.291 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:31:18.291 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:31:18.292 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:31:18.292 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:31:18.292 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:31:18.292 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:31:18.573 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:31:18.573+0000 7f726eb327c0 -1 Falling back to public interface 2024-03-25T23:31:18.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:31:19.600 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:31:19.600 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:31:19.601 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:31:19.601 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:31:19.601 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:31:19.601 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:31:19.918 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:31:20.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:31:20.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:31:20.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:31:20.920 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:31:20.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:31:20.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:31:21.296 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:31:21.520 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:31:21.521+0000 7f726eb327c0 -1 osd.2 0 log_to_monitors true 2024-03-25T23:31:22.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:31:22.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:31:22.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:31:22.298 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:31:22.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:31:22.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:31:22.687 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:31:23.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:31:23.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:31:23.689 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:31:23.690 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:31:23.690 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:31:23.690 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:31:24.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:31:25.011 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:31:25.012 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:31:25.012 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:31:25.012 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:31:25.012 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:31:25.013 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:31:25.336 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:31:26.339 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:31:26.339 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:31:26.339 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:31:26.339 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:31:26.340 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:31:26.340 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:31:26.809 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:31:27.811 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:31:27.811 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:31:27.811 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:31:27.812 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:31:27.812 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:31:27.812 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:31:28.351 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:31:29.353 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:31:29.353 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:31:29.353 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:31:29.354 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:31:29.354 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:31:29.354 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:31:29.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:31:30.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:31:30.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:31:30.778 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:31:30.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:31:30.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:31:30.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:31:31.171 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:31:32.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:31:32.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:31:32.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:31:32.173 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:31:32.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:31:32.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:31:32.557 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:31:33.559 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:31:33.559 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:31:33.559 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:31:33.559 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:31:33.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:31:33.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:31:33.954 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:31:34.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:31:34.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:31:34.956 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:31:34.957 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:31:34.957 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:31:34.957 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:31:35.317 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:31:36.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:31:36.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:31:36.319 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:31:36.320 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:31:36.320 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:31:36.320 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:31:36.794 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:31:37.795 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:31:37.796 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:31:37.796 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:31:37.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:31:37.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:31:37.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:31:38.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:31:39.154 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:31:39.154 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:31:39.155 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:31:39.155 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:31:39.155 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:31:39.155 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:31:39.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:31:40.791 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:31:40.791 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:31:40.792 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:31:40.792 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:31:40.792 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:31:40.792 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:31:41.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:31:42.259 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:31:42.259 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:31:42.259 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:31:42.259 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:31:42.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:31:42.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:31:42.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:31:43.718 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:31:43.718 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:31:43.718 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:31:43.719 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:31:43.719 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:31:43.719 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:31:44.637 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:31:45.639 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:31:45.639 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:31:45.639 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:31:45.640 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:31:45.640 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:31:45.640 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:31:46.213 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:31:47.216 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:31:47.216 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:31:47.216 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-25T23:31:47.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-25T23:31:47.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:31:47.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:31:47.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:31:48.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:31:48.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:31:48.735 INFO:tasks.workunit.client.0.smithi129.stdout:21 2024-03-25T23:31:48.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-25T23:31:48.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:31:48.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:31:49.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:31:50.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:31:50.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:31:50.263 INFO:tasks.workunit.client.0.smithi129.stdout:22 2024-03-25T23:31:50.264 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-25T23:31:50.264 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:31:50.264 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:31:50.581 INFO:tasks.workunit.client.0.smithi129.stdout:osd.2 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3084136331,v1:127.0.0.1:6819/3084136331] [v2:127.0.0.1:6820/3084136331,v1:127.0.0.1:6821/3084136331] exists,up fb6d380b-0822-4d14-bb7d-c99654850a39 2024-03-25T23:31:50.582 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:31:50.582 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:31:50.582 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:31:50.582 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:283: auto_repair_erasure_coded: create_rbd_pool 2024-03-25T23:31:50.582 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-25T23:31:51.186 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'rbd' does not exist 2024-03-25T23:31:51.203 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-25T23:31:51.203 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-25T23:31:51.717 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'rbd' created 2024-03-25T23:31:51.739 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T23:31:52.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-25T23:31:55.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:284: auto_repair_erasure_coded: wait_for_clean 2024-03-25T23:31:55.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:31:55.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:31:55.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:31:55.344 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:31:55.345 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:31:55.345 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:31:55.346 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:31:55.346 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:31:55.346 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:31:55.470 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:31:55.470 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:31:55.471 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:31:55.471 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:31:55.471 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:31:55.471 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:31:55.792 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:31:55.792 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:31:55.792 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-25T23:31:55.792 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:31:55.792 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:31:55.792 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:31:55.950 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836501 2024-03-25T23:31:55.950 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836501 2024-03-25T23:31:55.950 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501' 2024-03-25T23:31:55.950 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:31:55.950 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:31:56.091 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=68719476751 2024-03-25T23:31:56.091 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 68719476751 2024-03-25T23:31:56.091 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-68719476751' 2024-03-25T23:31:56.091 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:31:56.091 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:31:56.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855366 2024-03-25T23:31:56.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855366 2024-03-25T23:31:56.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-68719476751 2-150323855366' 2024-03-25T23:31:56.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:31:56.237 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836501 2024-03-25T23:31:56.237 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:31:56.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:31:56.238 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836501 2024-03-25T23:31:56.238 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:31:56.239 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836501 2024-03-25T23:31:56.239 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-03-25T23:31:56.240 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 21474836501 2024-03-25T23:31:56.240 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:31:56.548 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836501 2024-03-25T23:31:56.548 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:31:56.549 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-68719476751 2024-03-25T23:31:56.549 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:31:56.551 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:31:56.552 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-68719476751 2024-03-25T23:31:56.552 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:31:56.553 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 68719476751 2024-03-25T23:31:56.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=68719476751 2024-03-25T23:31:56.554 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 68719476751' 2024-03-25T23:31:56.554 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:31:56.875 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 68719476751 -lt 68719476751 2024-03-25T23:31:56.875 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:31:56.876 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-150323855366 2024-03-25T23:31:56.876 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:31:56.878 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:31:56.879 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-150323855366 2024-03-25T23:31:56.879 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:31:56.880 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855366 2024-03-25T23:31:56.880 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 150323855366' 2024-03-25T23:31:56.881 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 150323855366 2024-03-25T23:31:56.881 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:31:57.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855365 -lt 150323855366 2024-03-25T23:31:57.200 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:31:58.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:31:58.201 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:31:58.522 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855366 -lt 150323855366 2024-03-25T23:31:58.522 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:31:58.523 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:31:58.523 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:31:58.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-25T23:31:58.885 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:31:58.885 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:31:58.885 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:31:58.885 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:31:58.885 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:31:58.885 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:31:58.885 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:31:59.195 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T23:31:59.196 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:31:59.196 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:31:59.196 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:31:59.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-25T23:31:59.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:31:59.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:31:59.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: create_ec_pool ecpool true k=2 m=1 2024-03-25T23:31:59.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-25T23:31:59.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-25T23:31:59.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-03-25T23:31:59.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-25T23:31:59.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-03-25T23:32:00.208 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-25T23:32:00.208 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-25T23:32:01.281 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'ecpool' created 2024-03-25T23:32:01.304 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T23:32:02.306 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-03-25T23:32:02.306 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-03-25T23:32:02.786 INFO:tasks.workunit.client.0.smithi129.stderr:set pool 2 allow_ec_overwrites to true 2024-03-25T23:32:02.809 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-25T23:32:02.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:32:02.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:32:02.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:32:02.810 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:32:02.811 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:32:02.811 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:32:02.811 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:32:02.811 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:32:02.811 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:32:02.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:32:02.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:32:02.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:32:02.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:32:02.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:32:02.930 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:32:03.357 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:32:03.357 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:32:03.357 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-25T23:32:03.357 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:32:03.357 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:32:03.358 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:32:03.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836504 2024-03-25T23:32:03.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836504 2024-03-25T23:32:03.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504' 2024-03-25T23:32:03.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:32:03.499 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:32:03.691 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=68719476753 2024-03-25T23:32:03.691 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 68719476753 2024-03-25T23:32:03.691 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-68719476753' 2024-03-25T23:32:03.691 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:32:03.691 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:32:03.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855368 2024-03-25T23:32:03.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855368 2024-03-25T23:32:03.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-68719476753 2-150323855368' 2024-03-25T23:32:03.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:32:03.833 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836504 2024-03-25T23:32:03.833 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:32:03.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:32:03.835 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836504 2024-03-25T23:32:03.835 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:32:03.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836504 2024-03-25T23:32:03.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-03-25T23:32:03.836 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 21474836504 2024-03-25T23:32:03.836 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:32:04.130 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836504 2024-03-25T23:32:04.130 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:32:05.131 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:32:05.132 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:32:05.488 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836504 2024-03-25T23:32:05.488 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:32:05.488 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-68719476753 2024-03-25T23:32:05.488 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:32:05.490 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:32:05.490 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-68719476753 2024-03-25T23:32:05.491 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:32:05.492 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=68719476753 2024-03-25T23:32:05.492 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 68719476753' 2024-03-25T23:32:05.492 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 68719476753 2024-03-25T23:32:05.492 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:32:05.933 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 68719476753 -lt 68719476753 2024-03-25T23:32:05.933 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:32:05.934 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-150323855368 2024-03-25T23:32:05.934 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:32:05.937 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:32:05.937 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-150323855368 2024-03-25T23:32:05.937 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:32:05.938 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855368 2024-03-25T23:32:05.939 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 150323855368' 2024-03-25T23:32:05.939 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 150323855368 2024-03-25T23:32:05.939 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:32:06.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855368 -lt 150323855368 2024-03-25T23:32:06.257 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:32:06.257 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:32:06.258 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:32:06.682 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:32:06.682 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:32:06.682 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:32:06.682 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:32:06.682 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:32:06.683 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:32:06.683 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:32:06.683 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:32:06.997 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:32:06.997 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:32:06.998 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:32:06.998 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:32:07.372 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:32:07.372 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:32:07.372 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:32:07.372 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-25T23:32:07.372 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:290: auto_repair_erasure_coded: local payload=ABCDEF 2024-03-25T23:32:07.372 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:291: auto_repair_erasure_coded: echo ABCDEF 2024-03-25T23:32:07.372 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:292: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-25T23:32:07.861 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-03-25T23:32:07.861 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-03-25T23:32:07.861 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-25T23:32:07.862 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-03-25T23:32:07.862 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-03-25T23:32:07.862 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-25T23:32:07.863 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-25T23:32:07.863 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-03-25T23:32:08.185 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-03-25T23:32:08.185 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-03-25T23:32:08.185 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-03-25T23:32:08.524 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-25T23:32:08.524 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:32:08.524 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:32:08.524 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-25T23:32:08.525 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:32:08.525 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-25T23:32:08.525 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:32:08.525 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:32:08.525 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-25T23:32:08.525 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:32:08.525 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-25T23:32:08.525 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:32:08.526 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:32:08.526 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:32:08.526 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:32:08.526 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:32:08.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:32:08.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-25T23:32:08.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:32:08.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:32:08.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-25T23:32:08.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:32:08.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:32:08.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-25T23:32:11.125 INFO:tasks.workunit.client.0.smithi129.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-03-25T23:32:11.126 INFO:tasks.workunit.client.0.smithi129.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-03-25T23:32:11.126 INFO:tasks.workunit.client.0.smithi129.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-03-25T23:32:11.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:32:11.666 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:32:11.666 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:32:11.666 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-25T23:32:11.666 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:32:11.666 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:32:11.666 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:32:11.666 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:32:11.666 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:32:11.666 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:32:11.666 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-25T23:32:11.666 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-25T23:32:11.666 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:32:11.666 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:32:11.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:32:11.667 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:32:11.667 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:32:11.667 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:32:11.667 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:32:11.667 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:32:11.667 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:32:11.668 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:32:11.668 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:32:11.668 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:32:11.668 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:32:11.668 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:32:11.668 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:32:11.669 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:32:11.669 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:32:11.669 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:32:11.669 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:32:11.669 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-25T23:32:11.669 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-25T23:32:11.669 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-25T23:32:11.669 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.1 2024-03-25T23:32:11.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:32:11.670 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-25T23:32:11.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-25T23:32:11.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:32:11.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:32:11.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:32:11.707 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:32:11.708+0000 7feeb79d77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:32:11.709 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:32:11.708+0000 7feeb79d77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:32:11.711 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:32:11.712+0000 7feeb79d77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:32:11.990 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-25T23:32:11.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:32:11.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T23:32:11.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:32:11.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:32:11.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:32:11.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:32:11.991 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-25T23:32:11.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:32:11.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:32:12.252 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:32:12.252+0000 7feeb79d77c0 -1 Falling back to public interface 2024-03-25T23:32:12.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:32:13.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:32:13.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:32:13.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:32:13.299 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:32:13.300 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:32:13.300 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:32:13.604 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:32:14.606 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:32:14.606 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:32:14.606 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:32:14.607 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:32:14.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:32:14.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:32:14.936 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:32:14.936+0000 7feeb79d77c0 -1 osd.1 56 log_to_monitors true 2024-03-25T23:32:14.995 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:32:15.997 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:32:15.997 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:32:15.997 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:32:15.997 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:32:15.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:32:15.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:32:16.444 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1 up in weight 1 up_from 63 up_thru 63 down_at 57 last_clean_interval [16,56) [v2:127.0.0.1:6810/312200007,v1:127.0.0.1:6811/312200007] [v2:127.0.0.1:6812/312200007,v1:127.0.0.1:6813/312200007] exists,up e17236c7-d76e-42d7-ba66-309769fab2f8 2024-03-25T23:32:16.444 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:32:16.444 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:32:16.444 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:32:16.444 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:32:16.444 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:32:16.444 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:32:16.444 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:32:16.444 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:32:16.445 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:32:16.445 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:32:16.445 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:32:16.445 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:32:16.445 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:32:16.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:32:16.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:32:16.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:32:16.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:32:16.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:32:16.560 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:32:16.977 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:32:16.977 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:32:16.977 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-25T23:32:16.977 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:32:16.977 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:32:16.978 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:32:17.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836508 2024-03-25T23:32:17.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836508 2024-03-25T23:32:17.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508' 2024-03-25T23:32:17.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:32:17.232 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:32:17.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=270582939650 2024-03-25T23:32:17.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 270582939650 2024-03-25T23:32:17.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-270582939650' 2024-03-25T23:32:17.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:32:17.508 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:32:17.738 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855372 2024-03-25T23:32:17.738 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855372 2024-03-25T23:32:17.738 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-270582939650 2-150323855372' 2024-03-25T23:32:17.738 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:32:17.739 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:32:17.739 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836508 2024-03-25T23:32:17.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:32:17.741 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836508 2024-03-25T23:32:17.741 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:32:17.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836508 2024-03-25T23:32:17.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-03-25T23:32:17.742 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 21474836508 2024-03-25T23:32:17.742 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:32:18.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836508 2024-03-25T23:32:18.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:32:19.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:32:19.168 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:32:19.490 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836508 2024-03-25T23:32:19.491 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:32:19.491 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-270582939650 2024-03-25T23:32:19.491 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:32:19.491 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:32:19.491 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-270582939650 2024-03-25T23:32:19.491 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:32:19.492 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=270582939650 2024-03-25T23:32:19.492 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 270582939650' 2024-03-25T23:32:19.492 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 270582939650 2024-03-25T23:32:19.493 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:32:19.818 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 270582939650 -lt 270582939650 2024-03-25T23:32:19.818 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:32:19.819 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:32:19.819 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-150323855372 2024-03-25T23:32:19.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:32:19.821 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-150323855372 2024-03-25T23:32:19.821 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:32:19.822 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855372 2024-03-25T23:32:19.822 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 150323855372' 2024-03-25T23:32:19.822 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 150323855372 2024-03-25T23:32:19.823 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:32:20.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855372 -lt 150323855372 2024-03-25T23:32:20.238 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:32:20.238 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:32:20.239 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:32:20.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:32:20.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:32:20.631 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:32:20.631 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:32:20.631 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:32:20.631 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:32:20.632 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:32:20.632 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:32:20.968 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T23:32:20.969 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:32:20.969 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:32:20.969 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:32:21.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-25T23:32:21.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-25T23:32:21.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-25T23:32:21.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-25T23:32:21.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:32:21.380 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-25T23:32:21.480 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:32:21.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:32:21.481 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:32:21.482 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:32:21.482 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:32:21.482 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:32:21.482 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:32:21.482 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:32:21.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:32:21.798 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:32:21.798 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:32:21.798 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:32:22.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:32:22.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:32:22.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:32:22.166 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2024-03-25T23:32:22.166 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-25T23:32:22.166 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-25T23:32:22.167 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-25T23:32:22.167 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-25T23:32:22.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: local pgid=2.0 2024-03-25T23:32:22.458 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2024-03-25T23:32:22.459 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:32:22.459 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:32:22.459 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:32:22.459 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:32:22.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: wait_for_scrub 2.0 2024-03-25T23:32:07.705104+0000 2024-03-25T23:32:22.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-25T23:32:22.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-25T23:32:07.705104+0000 2024-03-25T23:32:22.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-25T23:32:22.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-25T23:32:22.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:32:22.753 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:32:22.753 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:32:22.753 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:32:22.753 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:32:22.753 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:32:23.056 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:32:07.705104+0000 '>' 2024-03-25T23:32:07.705104+0000 2024-03-25T23:32:23.056 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:32:24.058 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:32:24.058 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:32:24.058 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:32:24.059 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:32:24.059 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:32:24.059 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:32:24.059 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:32:24.375 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:32:07.705104+0000 '>' 2024-03-25T23:32:07.705104+0000 2024-03-25T23:32:24.375 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:32:25.376 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:32:25.376 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:32:25.377 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:32:25.377 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:32:25.377 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:32:25.378 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:32:25.378 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:32:25.699 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:32:07.705104+0000 '>' 2024-03-25T23:32:07.705104+0000 2024-03-25T23:32:25.699 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:32:26.701 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:32:26.701 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:32:26.702 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:32:26.702 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:32:26.702 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:32:26.702 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:32:26.703 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:32:27.019 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:32:21.702242+0000 '>' 2024-03-25T23:32:07.705104+0000 2024-03-25T23:32:27.019 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-25T23:32:27.019 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:300: auto_repair_erasure_coded: wait_for_clean 2024-03-25T23:32:27.020 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:32:27.020 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:32:27.020 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:32:27.020 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:32:27.020 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:32:27.021 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:32:27.021 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:32:27.021 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:32:27.021 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:32:27.130 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:32:27.130 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:32:27.130 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:32:27.130 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:32:27.131 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:32:27.131 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:32:27.448 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:32:27.448 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:32:27.448 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-25T23:32:27.448 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:32:27.448 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:32:27.449 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:32:27.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836511 2024-03-25T23:32:27.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836511 2024-03-25T23:32:27.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511' 2024-03-25T23:32:27.597 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:32:27.597 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:32:27.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=270582939653 2024-03-25T23:32:27.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 270582939653 2024-03-25T23:32:27.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511 1-270582939653' 2024-03-25T23:32:27.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:32:27.757 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:32:27.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855375 2024-03-25T23:32:27.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855375 2024-03-25T23:32:27.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511 1-270582939653 2-150323855375' 2024-03-25T23:32:27.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:32:27.927 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836511 2024-03-25T23:32:27.927 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:32:27.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:32:27.930 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836511 2024-03-25T23:32:27.930 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:32:27.931 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836511 2024-03-25T23:32:27.931 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2024-03-25T23:32:27.931 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 21474836511 2024-03-25T23:32:27.932 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:32:28.242 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836509 -lt 21474836511 2024-03-25T23:32:28.242 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:32:29.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:32:29.244 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:32:29.563 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836511 -lt 21474836511 2024-03-25T23:32:29.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:32:29.564 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-270582939653 2024-03-25T23:32:29.565 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:32:29.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:32:29.567 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-270582939653 2024-03-25T23:32:29.568 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:32:29.569 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=270582939653 2024-03-25T23:32:29.569 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 270582939653 2024-03-25T23:32:29.569 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 270582939653' 2024-03-25T23:32:29.570 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:32:29.886 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 270582939653 -lt 270582939653 2024-03-25T23:32:29.886 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:32:29.886 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-150323855375 2024-03-25T23:32:29.887 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:32:29.888 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:32:29.889 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:32:29.889 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-150323855375 2024-03-25T23:32:29.890 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855375 2024-03-25T23:32:29.890 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 150323855375 2024-03-25T23:32:29.891 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 150323855375' 2024-03-25T23:32:29.891 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:32:30.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855375 -lt 150323855375 2024-03-25T23:32:30.230 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:32:30.230 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:32:30.230 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:32:30.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:32:30.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:32:30.608 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:32:30.608 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:32:30.608 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:32:30.608 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:32:30.608 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:32:30.608 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:32:30.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:32:30.920 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:32:30.921 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:32:30.921 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:32:31.295 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:32:31.295 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:32:31.295 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:32:31.295 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-03-25T23:32:31.295 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-03-25T23:32:31.295 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-25T23:32:31.295 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-03-25T23:32:31.295 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-03-25T23:32:31.295 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-25T23:32:31.296 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-03-25T23:32:31.296 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-25T23:32:31.608 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-03-25T23:32:31.608 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-03-25T23:32:31.609 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-03-25T23:32:31.971 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-25T23:32:31.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:32:31.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:32:31.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-25T23:32:31.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:32:31.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-25T23:32:31.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:32:31.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:32:31.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-25T23:32:31.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:32:31.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-25T23:32:31.972 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:32:31.972 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:32:31.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:32:31.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:32:31.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:32:32.282 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:32:32.282 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-25T23:32:32.282 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:32:32.282 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:32:32.282 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-25T23:32:32.282 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:32:32.282 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:32:32.282 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-25T23:32:32.435 INFO:tasks.workunit.client.0.smithi129.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-03-25T23:32:32.436 INFO:tasks.workunit.client.0.smithi129.stdout:_ 2024-03-25T23:32:32.436 INFO:tasks.workunit.client.0.smithi129.stdout:hinfo_key 2024-03-25T23:32:32.436 INFO:tasks.workunit.client.0.smithi129.stdout:snapset 2024-03-25T23:32:32.436 INFO:tasks.workunit.client.0.smithi129.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-03-25T23:32:32.726 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:32:32.726 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:32:32.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:32:32.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-25T23:32:32.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:32:32.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:32:32.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:32:32.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:32:32.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:32:32.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:32:32.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-25T23:32:32.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-25T23:32:32.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:32:32.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:32:32.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:32:32.727 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:32:32.728 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:32:32.728 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:32:32.729 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:32:32.729 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:32:32.729 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:32:32.729 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:32:32.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:32:32.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:32:32.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:32:32.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:32:32.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:32:32.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:32:32.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:32:32.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:32:32.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:32:32.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-25T23:32:32.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-25T23:32:32.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-25T23:32:32.731 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.1 2024-03-25T23:32:32.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-25T23:32:32.731 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-25T23:32:32.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-25T23:32:32.733 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:32:32.733 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:32:32.733 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:32:32.768 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:32:32.768+0000 7f45d56697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:32:32.823 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:32:32.824+0000 7f45d56697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:32:32.825 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:32:32.824+0000 7f45d56697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:32:33.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-25T23:32:33.146 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:32:33.146 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T23:32:33.146 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:32:33.146 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:32:33.146 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:32:33.146 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:32:33.146 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-25T23:32:33.146 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:32:33.146 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:32:33.376 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:32:33.376+0000 7f45d56697c0 -1 Falling back to public interface 2024-03-25T23:32:33.600 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:32:34.602 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:32:34.602 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:32:34.602 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:32:34.602 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:32:34.603 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:32:34.603 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:32:34.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:32:35.924 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:32:35.924 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:32:35.924 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:32:35.924 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:32:35.924 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:32:35.925 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:32:36.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:32:36.399 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:32:36.400+0000 7f45d56697c0 -1 osd.1 73 log_to_monitors true 2024-03-25T23:32:37.239 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:32:37.239 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:32:37.239 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:32:37.239 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:32:37.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:32:37.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:32:37.797 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1 up in weight 1 up_from 80 up_thru 63 down_at 75 last_clean_interval [63,73) [v2:127.0.0.1:6810/855777791,v1:127.0.0.1:6811/855777791] [v2:127.0.0.1:6812/855777791,v1:127.0.0.1:6813/855777791] exists,up e17236c7-d76e-42d7-ba66-309769fab2f8 2024-03-25T23:32:37.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:32:37.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:32:37.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:32:37.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:32:37.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:32:37.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:32:37.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:32:37.798 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:32:37.798 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:32:37.798 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:32:37.798 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:32:37.798 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:32:37.799 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:32:37.916 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:32:37.916 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:32:37.916 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:32:37.916 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:32:37.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:32:37.917 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:32:38.244 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:32:38.244 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:32:38.244 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-25T23:32:38.244 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:32:38.244 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:32:38.244 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:32:38.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836514 2024-03-25T23:32:38.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836514 2024-03-25T23:32:38.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514' 2024-03-25T23:32:38.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:32:38.528 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:32:38.687 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383682 2024-03-25T23:32:38.687 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383682 2024-03-25T23:32:38.687 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514 1-343597383682' 2024-03-25T23:32:38.687 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:32:38.687 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:32:38.828 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855378 2024-03-25T23:32:38.828 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855378 2024-03-25T23:32:38.828 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514 1-343597383682 2-150323855378' 2024-03-25T23:32:38.828 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:32:38.829 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836514 2024-03-25T23:32:38.829 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:32:38.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:32:38.830 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836514 2024-03-25T23:32:38.830 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:32:38.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836514 2024-03-25T23:32:38.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836514' 2024-03-25T23:32:38.831 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 21474836514 2024-03-25T23:32:38.832 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:32:39.208 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836513 -lt 21474836514 2024-03-25T23:32:39.208 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:32:40.210 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:32:40.211 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:32:40.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836514 -lt 21474836514 2024-03-25T23:32:40.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:32:40.533 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-343597383682 2024-03-25T23:32:40.534 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:32:40.535 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:32:40.536 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-343597383682 2024-03-25T23:32:40.536 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:32:40.537 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383682 2024-03-25T23:32:40.537 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 343597383682' 2024-03-25T23:32:40.537 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 343597383682 2024-03-25T23:32:40.538 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:32:40.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383682 -lt 343597383682 2024-03-25T23:32:40.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:32:40.869 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:32:40.870 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-150323855378 2024-03-25T23:32:40.871 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:32:40.871 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-150323855378 2024-03-25T23:32:40.872 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:32:40.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855378 2024-03-25T23:32:40.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 150323855378' 2024-03-25T23:32:40.873 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 150323855378 2024-03-25T23:32:40.873 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:32:41.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855379 -lt 150323855378 2024-03-25T23:32:41.173 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:32:41.173 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:32:41.174 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:32:41.551 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:32:41.552 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:32:41.552 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:32:41.552 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:32:41.552 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:32:41.552 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:32:41.553 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:32:41.553 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:32:41.866 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T23:32:41.867 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:32:41.868 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:32:41.868 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:32:42.221 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-25T23:32:42.221 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-25T23:32:42.221 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-25T23:32:42.221 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-25T23:32:42.221 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:32:42.221 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-25T23:32:42.322 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:32:42.322 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:32:42.323 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:32:42.323 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:32:42.323 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:32:42.323 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:32:42.324 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:32:42.324 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:32:42.658 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:32:42.659 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:32:42.659 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:32:42.660 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:32:43.012 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:32:43.012 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:32:43.012 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:32:43.012 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-25T23:32:43.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:305: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-25T23:32:43.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-25T23:32:43.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-25T23:32:43.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T23:32:43.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-25T23:32:43.048 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:32:43.048 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:32:43.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:32:43.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:32:43.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:32:43.355 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:32:43.355 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T23:32:43.357 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T23:32:43.357 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T23:32:43.358 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T23:32:43.358 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T23:32:43.359 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T23:32:43.360 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:32:43.360 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T23:32:43.360 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T23:32:43.361 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:32:43.361 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T23:32:43.362 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T23:32:43.363 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-25T23:32:43.386 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T23:32:43.386 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:32:43.386 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:32:43.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-25T23:32:43.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T23:32:43.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T23:32:43.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-25T23:32:43.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-25T23:32:43.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-25T23:32:43.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-25T23:32:43.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-25T23:32:43.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T23:32:43.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-25T23:32:43.389 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:32:43.389 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:32:43.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:32:43.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:32:43.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:32:43.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:32:43.391 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T23:32:43.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T23:32:43.392 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T23:32:43.393 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T23:32:43.393 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T23:32:43.393 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T23:32:43.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:32:43.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T23:32:43.395 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T23:32:43.395 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:32:43.396 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T23:32:43.397 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T23:32:43.397 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-25T23:32:43.398 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T23:32:43.398 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:32:43.399 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:32:43.399 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-25T23:32:43.400 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T23:32:43.400 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T23:32:43.400 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-25T23:32:43.402 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-25T23:32:43.402 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:32:43.402 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:32:43.402 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61636 2024-03-25T23:32:43.405 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-25T23:32:43.405 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-25T23:32:43.405 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-25T23:32:43.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-25T23:32:43.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_jerasure_appends td/osd-scrub-repair 2024-03-25T23:32:43.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:815: TEST_corrupt_and_repair_jerasure_appends: corrupt_and_repair_jerasure td/osd-scrub-repair false 2024-03-25T23:32:43.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:798: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2024-03-25T23:32:43.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:799: corrupt_and_repair_jerasure: local allow_overwrites=false 2024-03-25T23:32:43.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:800: corrupt_and_repair_jerasure: local poolname=ecpool 2024-03-25T23:32:43.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:802: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2024-03-25T23:32:43.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-25T23:32:43.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-25T23:32:43.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-25T23:32:43.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-25T23:32:43.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-25T23:32:43.407 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-25T23:32:43.901 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-25T23:32:43.901 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:32:43.901 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:32:43.902 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:32:43.902 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:32:43.902 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:32:43.902 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:32:43.902 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-25T23:32:44.104 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-25T23:32:44.106 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-25T23:32:44.106 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T23:32:44.106 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T23:32:44.106 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-25T23:32:44.107 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-25T23:32:44.107 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T23:32:44.107 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T23:32:44.107 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T23:32:44.108 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T23:32:44.108 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:32:44.108 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:32:44.108 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-25T23:32:44.108 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T23:32:44.108 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get fsid 2024-03-25T23:32:44.223 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-25T23:32:44.224 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T23:32:44.224 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T23:32:44.224 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-25T23:32:44.224 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-25T23:32:44.225 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T23:32:44.225 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T23:32:44.225 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T23:32:44.225 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T23:32:44.225 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:32:44.225 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:32:44.226 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-25T23:32:44.226 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T23:32:44.226 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get mon_host 2024-03-25T23:32:44.334 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:803: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2024-03-25T23:32:44.334 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-25T23:32:44.334 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-25T23:32:44.335 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-25T23:32:44.335 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-25T23:32:44.335 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-25T23:32:44.335 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-25T23:32:45.223 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-25T23:32:45.223 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:32:45.224 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:32:45.224 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:32:45.224 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:32:45.224 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:32:45.224 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:32:45.224 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T23:32:45.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T23:32:45.261 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: seq 0 3 2024-03-25T23:32:45.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-03-25T23:32:45.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 0 2024-03-25T23:32:45.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:32:45.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:32:45.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-25T23:32:45.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:32:45.266 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-25T23:32:45.266 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:32:45.266 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:32:45.266 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:32:45.266 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:32:45.266 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-25T23:32:45.266 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-25T23:32:45.266 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:32:45.266 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:32:45.266 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:32:45.266 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:32:45.266 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:32:45.266 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:32:45.266 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:32:45.266 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:32:45.267 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:32:45.267 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:32:45.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:32:45.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:32:45.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:32:45.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:32:45.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:32:45.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:32:45.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:32:45.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:32:45.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:32:45.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:32:45.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:32:45.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T23:32:45.268 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-25T23:32:45.268 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:32:45.268 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0ef765fa-1e1d-41a9-bbe2-b544ea887ed2 2024-03-25T23:32:45.268 INFO:tasks.workunit.client.0.smithi129.stdout:add osd0 0ef765fa-1e1d-41a9-bbe2-b544ea887ed2 2024-03-25T23:32:45.269 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0ef765fa-1e1d-41a9-bbe2-b544ea887ed2' 2024-03-25T23:32:45.269 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:32:45.285 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCdCQJmkT0rERAAoh0YeuNyOtHWdOM5QbMYIw== 2024-03-25T23:32:45.286 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCdCQJmkT0rERAAoh0YeuNyOtHWdOM5QbMYIw=="}' 2024-03-25T23:32:45.286 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0ef765fa-1e1d-41a9-bbe2-b544ea887ed2 -i td/osd-scrub-repair/0/new.json 2024-03-25T23:32:45.557 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:32:45.573 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-25T23:32:45.575 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCdCQJmkT0rERAAoh0YeuNyOtHWdOM5QbMYIw== --osd-uuid 0ef765fa-1e1d-41a9-bbe2-b544ea887ed2 2024-03-25T23:32:45.605 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:32:45.604+0000 7f9a6dceb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:32:45.624 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:32:45.624+0000 7f9a6dceb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:32:45.625 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:32:45.624+0000 7f9a6dceb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:32:45.625 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:32:45.624+0000 7f9a6dceb7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-25T23:32:45.626 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:32:45.624+0000 7f9a6dceb7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-25T23:32:45.626 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:32:45.624+0000 7f9a6dceb7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-25T23:32:45.724 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:32:45.724+0000 7f9a6dceb7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-25T23:32:52.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-25T23:32:52.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:32:52.071 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-25T23:32:52.071 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd0 key to auth repository 2024-03-25T23:32:52.071 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:32:52.453 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.0 2024-03-25T23:32:52.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-25T23:32:52.454 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:32:52.454 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:32:52.454 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:32:52.454 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:32:52.484 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:32:52.484+0000 7f596b4057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:32:52.486 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:32:52.488+0000 7f596b4057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:32:52.488 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:32:52.488+0000 7f596b4057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:32:52.765 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-25T23:32:52.766 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:32:52.766 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T23:32:52.766 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:32:52.766 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:32:52.766 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:32:52.766 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:32:52.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:32:52.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:32:52.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:32:53.040 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:32:53.040+0000 7f596b4057c0 -1 Falling back to public interface 2024-03-25T23:32:53.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:32:54.126 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:32:54.127 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:32:54.127 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:32:54.127 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:32:54.128 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:32:54.128 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:32:54.444 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:32:55.446 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:32:55.446 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:32:55.446 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:32:55.447 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:32:55.447 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:32:55.447 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:32:55.686 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:32:55.688+0000 7f596b4057c0 -1 osd.0 0 log_to_monitors true 2024-03-25T23:32:55.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:32:56.804 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:32:56.804 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:32:56.804 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:32:56.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:32:56.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:32:56.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:32:57.127 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:32:58.129 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:32:58.129 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:32:58.129 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:32:58.129 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:32:58.130 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:32:58.130 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:32:58.525 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:32:59.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:32:59.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:32:59.527 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:32:59.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:32:59.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:32:59.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:32:59.847 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:33:00.849 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:33:00.849 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:33:00.849 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:33:00.850 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:33:00.850 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:33:00.850 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:33:01.167 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:33:02.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:33:02.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:33:02.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:33:02.169 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:33:02.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:33:02.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:33:02.542 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:33:03.544 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:33:03.544 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:33:03.544 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:33:03.545 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:33:03.545 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:33:03.545 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:33:03.864 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:33:04.865 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:33:04.865 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:33:04.865 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:33:04.866 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:33:04.866 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:33:04.866 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:33:05.181 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:33:06.183 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:33:06.184 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:33:06.184 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:33:06.185 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:33:06.185 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:33:06.185 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:33:06.566 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:33:07.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:33:07.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:33:07.569 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:33:07.569 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:33:07.569 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:33:07.569 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:33:07.880 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:33:08.882 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:33:08.882 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:33:08.883 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:33:08.883 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:33:08.883 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:33:08.883 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:33:09.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:33:10.191 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:33:10.191 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:33:10.191 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:33:10.191 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:33:10.192 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:33:10.192 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:33:10.786 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:33:11.790 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:33:11.790 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:33:11.790 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:33:11.791 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:33:11.791 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:33:11.791 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:33:12.673 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:33:13.675 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:33:13.675 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:33:13.675 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:33:13.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:33:13.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:33:13.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:33:14.476 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:33:15.478 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:33:15.478 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:33:15.478 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:33:15.479 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:33:15.479 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:33:15.479 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:33:16.287 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:33:17.289 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:33:17.289 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:33:17.290 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:33:17.290 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:33:17.291 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:33:17.291 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:33:17.936 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:33:18.938 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:33:18.938 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:33:18.939 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:33:18.939 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:33:18.939 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:33:18.939 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:33:19.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:33:20.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:33:20.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:33:20.346 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:33:20.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:33:20.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:33:20.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:33:20.700 INFO:tasks.workunit.client.0.smithi129.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/436571200,v1:127.0.0.1:6803/436571200] [v2:127.0.0.1:6804/436571200,v1:127.0.0.1:6805/436571200] exists,up 0ef765fa-1e1d-41a9-bbe2-b544ea887ed2 2024-03-25T23:33:20.702 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:33:20.702 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:33:20.702 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:33:20.702 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-03-25T23:33:20.702 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 1 2024-03-25T23:33:20.702 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:33:20.702 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:33:20.702 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-25T23:33:20.702 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:33:20.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:33:20.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:33:20.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:33:20.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:33:20.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:33:20.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-25T23:33:20.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-25T23:33:20.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:33:20.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:33:20.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:33:20.703 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:33:20.703 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:33:20.703 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:33:20.703 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:33:20.704 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:33:20.704 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:33:20.705 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:33:20.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:33:20.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:33:20.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:33:20.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:33:20.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:33:20.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:33:20.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:33:20.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:33:20.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:33:20.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:33:20.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:33:20.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T23:33:20.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-25T23:33:20.707 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:33:20.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0a1d267b-774d-4fda-8cc0-7d86cd87d221 2024-03-25T23:33:20.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 0a1d267b-774d-4fda-8cc0-7d86cd87d221' 2024-03-25T23:33:20.707 INFO:tasks.workunit.client.0.smithi129.stdout:add osd1 0a1d267b-774d-4fda-8cc0-7d86cd87d221 2024-03-25T23:33:20.707 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:33:20.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDACQJmvK9+KxAA72keT4354dYCqQdC35F/5g== 2024-03-25T23:33:20.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDACQJmvK9+KxAA72keT4354dYCqQdC35F/5g=="}' 2024-03-25T23:33:20.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0a1d267b-774d-4fda-8cc0-7d86cd87d221 -i td/osd-scrub-repair/1/new.json 2024-03-25T23:33:21.225 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:33:21.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-25T23:33:21.244 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDACQJmvK9+KxAA72keT4354dYCqQdC35F/5g== --osd-uuid 0a1d267b-774d-4fda-8cc0-7d86cd87d221 2024-03-25T23:33:21.279 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:33:21.279+0000 7f8e35d5f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:33:21.317 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:33:21.315+0000 7f8e35d5f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:33:21.318 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:33:21.319+0000 7f8e35d5f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:33:21.318 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:33:21.319+0000 7f8e35d5f7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-25T23:33:21.319 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:33:21.319+0000 7f8e35d5f7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-25T23:33:21.319 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:33:21.319+0000 7f8e35d5f7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-25T23:33:21.393 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:33:21.395+0000 7f8e35d5f7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-25T23:33:28.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-25T23:33:28.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:33:28.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-25T23:33:28.086 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd1 key to auth repository 2024-03-25T23:33:28.087 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:33:28.481 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.1 2024-03-25T23:33:28.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-25T23:33:28.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:33:28.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:33:28.482 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:33:28.482 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:33:28.513 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:33:28.511+0000 7f7fbe7877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:33:28.521 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:33:28.519+0000 7f7fbe7877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:33:28.522 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:33:28.523+0000 7f7fbe7877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:33:28.801 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-25T23:33:28.801 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:33:28.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T23:33:28.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:33:28.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:33:28.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:33:28.802 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:33:28.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:33:28.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:33:28.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:33:29.081 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:33:29.079+0000 7f7fbe7877c0 -1 Falling back to public interface 2024-03-25T23:33:29.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:33:30.144 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:33:30.144 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:33:30.144 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:33:30.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:33:30.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:33:30.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:33:30.447 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:33:31.448 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:33:31.449 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:33:31.449 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:33:31.449 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:33:31.449 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:33:31.450 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:33:31.689 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:33:31.687+0000 7f7fbe7877c0 -1 osd.1 0 log_to_monitors true 2024-03-25T23:33:31.847 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:33:32.849 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:33:32.850 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:33:32.850 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:33:32.851 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:33:32.851 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:33:32.851 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:33:33.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:33:34.163 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:33:34.163 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:33:34.164 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:33:34.164 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:33:34.164 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:33:34.164 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:33:34.642 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:33:35.644 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:33:35.644 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:33:35.645 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:33:35.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:33:35.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:33:35.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:33:36.029 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:33:37.031 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:33:37.031 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:33:37.031 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:33:37.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:33:37.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:33:37.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:33:37.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:33:38.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:33:38.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:33:38.346 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:33:38.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:33:38.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:33:38.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:33:38.723 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:33:39.725 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:33:39.725 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:33:39.725 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:33:39.726 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:33:39.726 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:33:39.726 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:33:40.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:33:41.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:33:41.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:33:41.038 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:33:41.039 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:33:41.039 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:33:41.039 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:33:41.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:33:42.351 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:33:42.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:33:42.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:33:42.352 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:33:42.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:33:42.353 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:33:42.785 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:33:43.787 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:33:43.787 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:33:43.787 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:33:43.788 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:33:43.788 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:33:43.788 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:33:44.100 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:33:45.101 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:33:45.101 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:33:45.102 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:33:45.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:33:45.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:33:45.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:33:45.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:33:46.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:33:46.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:33:46.616 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:33:46.616 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:33:46.616 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:33:46.616 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:33:47.133 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:33:48.134 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:33:48.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:33:48.135 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:33:48.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:33:48.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:33:48.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:33:48.763 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:33:49.765 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:33:49.765 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:33:49.765 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:33:49.766 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:33:49.766 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:33:49.766 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:33:50.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:33:51.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:33:51.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:33:51.262 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:33:51.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:33:51.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:33:51.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:33:52.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:33:53.134 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:33:53.134 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:33:53.134 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:33:53.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:33:53.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:33:53.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:33:53.610 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:33:54.614 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:33:54.614 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:33:54.614 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:33:54.615 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:33:54.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:33:54.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:33:55.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:33:56.122 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:33:56.122 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:33:56.122 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:33:56.122 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:33:56.122 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:33:56.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:33:56.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:33:57.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:33:57.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:33:57.670 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-25T23:33:57.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-25T23:33:57.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:33:57.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:33:58.014 INFO:tasks.workunit.client.0.smithi129.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/95229508,v1:127.0.0.1:6811/95229508] [v2:127.0.0.1:6812/95229508,v1:127.0.0.1:6813/95229508] exists,up 0a1d267b-774d-4fda-8cc0-7d86cd87d221 2024-03-25T23:33:58.018 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:33:58.018 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:33:58.018 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:33:58.018 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-03-25T23:33:58.018 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 2 2024-03-25T23:33:58.018 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:33:58.019 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:33:58.019 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-25T23:33:58.019 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:33:58.019 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-25T23:33:58.019 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:33:58.019 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:33:58.019 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:33:58.019 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:33:58.019 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-25T23:33:58.019 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-25T23:33:58.019 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:33:58.020 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:33:58.020 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:33:58.020 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:33:58.020 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:33:58.020 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:33:58.020 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:33:58.020 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:33:58.021 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:33:58.021 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:33:58.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:33:58.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:33:58.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:33:58.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:33:58.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:33:58.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:33:58.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:33:58.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:33:58.022 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:33:58.022 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:33:58.022 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:33:58.022 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T23:33:58.022 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-25T23:33:58.023 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:33:58.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c816e86b-623d-482b-83dd-84132fd7b4ae 2024-03-25T23:33:58.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 c816e86b-623d-482b-83dd-84132fd7b4ae' 2024-03-25T23:33:58.023 INFO:tasks.workunit.client.0.smithi129.stdout:add osd2 c816e86b-623d-482b-83dd-84132fd7b4ae 2024-03-25T23:33:58.023 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:33:58.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDmCQJmDNBsAhAAo4ms7rD7cR7pmhid0q/7Pw== 2024-03-25T23:33:58.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDmCQJmDNBsAhAAo4ms7rD7cR7pmhid0q/7Pw=="}' 2024-03-25T23:33:58.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c816e86b-623d-482b-83dd-84132fd7b4ae -i td/osd-scrub-repair/2/new.json 2024-03-25T23:33:58.490 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:33:58.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-25T23:33:58.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDmCQJmDNBsAhAAo4ms7rD7cR7pmhid0q/7Pw== --osd-uuid c816e86b-623d-482b-83dd-84132fd7b4ae 2024-03-25T23:33:58.544 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:33:58.542+0000 7ff4bef437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:33:58.583 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:33:58.582+0000 7ff4bef437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:33:58.584 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:33:58.582+0000 7ff4bef437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:33:58.584 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:33:58.582+0000 7ff4bef437c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-25T23:33:58.584 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:33:58.582+0000 7ff4bef437c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-25T23:33:58.584 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:33:58.582+0000 7ff4bef437c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-25T23:33:58.665 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:33:58.666+0000 7ff4bef437c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-25T23:34:05.204 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-25T23:34:05.204 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:34:05.206 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd2 key to auth repository 2024-03-25T23:34:05.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-25T23:34:05.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:34:05.598 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.2 2024-03-25T23:34:05.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-25T23:34:05.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:34:05.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:34:05.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:34:05.600 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:34:05.631 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:34:05.630+0000 7f1d685437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:34:05.663 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:34:05.662+0000 7f1d685437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:34:05.665 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:34:05.662+0000 7f1d685437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:34:05.939 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-25T23:34:05.939 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:34:05.939 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-25T23:34:05.939 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:34:05.939 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:34:05.940 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:34:05.940 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:34:05.940 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:34:05.940 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:34:05.940 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:34:06.220 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:34:06.218+0000 7f1d685437c0 -1 Falling back to public interface 2024-03-25T23:34:06.277 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:34:07.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:34:07.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:34:07.279 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:34:07.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:34:07.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:34:07.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:34:07.589 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:34:08.591 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:34:08.591 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:34:08.592 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:34:08.592 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:34:08.593 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:34:08.593 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:34:08.877 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:34:08.874+0000 7f1d685437c0 -1 osd.2 0 log_to_monitors true 2024-03-25T23:34:08.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:34:09.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:34:09.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:34:09.918 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:34:09.918 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:34:09.918 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:34:09.918 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:34:10.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:34:11.312 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:34:11.312 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:34:11.313 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:34:11.313 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:34:11.313 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:34:11.313 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:34:11.639 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:34:12.640 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:34:12.641 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:34:12.641 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:34:12.641 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:34:12.642 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:34:12.642 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:34:12.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:34:13.961 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:34:13.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:34:13.962 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:34:13.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:34:13.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:34:13.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:34:14.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:34:15.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:34:15.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:34:15.263 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:34:15.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:34:15.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:34:15.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:34:15.621 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:34:16.622 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:34:16.622 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:34:16.623 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:34:16.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:34:16.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:34:16.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:34:17.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:34:18.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:34:18.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:34:18.097 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:34:18.098 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:34:18.098 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:34:18.098 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:34:18.448 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:34:19.450 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:34:19.450 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:34:19.451 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:34:19.451 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:34:19.451 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:34:19.451 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:34:19.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:34:20.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:34:20.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:34:20.833 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:34:20.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:34:20.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:34:20.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:34:21.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:34:22.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:34:22.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:34:22.141 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:34:22.141 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:34:22.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:34:22.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:34:22.465 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:34:23.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:34:23.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:34:23.467 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:34:23.468 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:34:23.468 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:34:23.468 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:34:24.183 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:34:25.185 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:34:25.185 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:34:25.186 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:34:25.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:34:25.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:34:25.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:34:25.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:34:26.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:34:26.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:34:26.838 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:34:26.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:34:26.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:34:26.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:34:27.737 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:34:28.739 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:34:28.739 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:34:28.739 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:34:28.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:34:28.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:34:28.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:34:29.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:34:30.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:34:30.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:34:30.581 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:34:30.582 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:34:30.582 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:34:30.582 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:34:31.306 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:34:32.308 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:34:32.308 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:34:32.308 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:34:32.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:34:32.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:34:32.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:34:32.947 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:34:33.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:34:33.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:34:33.949 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:34:33.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:34:33.950 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:34:33.950 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:34:34.476 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:34:35.478 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:34:35.478 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:34:35.479 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-25T23:34:35.480 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-25T23:34:35.480 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:34:35.480 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:34:35.820 INFO:tasks.workunit.client.0.smithi129.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/855079924,v1:127.0.0.1:6819/855079924] [v2:127.0.0.1:6820/855079924,v1:127.0.0.1:6821/855079924] exists,up c816e86b-623d-482b-83dd-84132fd7b4ae 2024-03-25T23:34:35.824 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:34:35.824 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:34:35.824 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:34:35.824 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-03-25T23:34:35.824 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 3 2024-03-25T23:34:35.824 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:34:35.824 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:34:35.824 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-03-25T23:34:35.825 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:34:35.825 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-03-25T23:34:35.825 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:34:35.825 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:34:35.825 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:34:35.825 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:34:35.825 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-25T23:34:35.825 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-25T23:34:35.825 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:34:35.825 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:34:35.825 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:34:35.825 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:34:35.825 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:34:35.826 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:34:35.826 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:34:35.826 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:34:35.826 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:34:35.826 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:34:35.826 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:34:35.826 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:34:35.826 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:34:35.826 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:34:35.826 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:34:35.826 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:34:35.826 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:34:35.826 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:34:35.826 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:34:35.827 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:34:35.827 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:34:35.827 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T23:34:35.827 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-03-25T23:34:35.827 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:34:35.827 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=614c4a39-b13c-4192-ba13-0218e9f2bcd7 2024-03-25T23:34:35.828 INFO:tasks.workunit.client.0.smithi129.stdout:add osd3 614c4a39-b13c-4192-ba13-0218e9f2bcd7 2024-03-25T23:34:35.828 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 614c4a39-b13c-4192-ba13-0218e9f2bcd7' 2024-03-25T23:34:35.828 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:34:35.848 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQALCgJmeEC4MhAA5aZ6GPLQFpr7swAu9hVO1w== 2024-03-25T23:34:35.848 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQALCgJmeEC4MhAA5aZ6GPLQFpr7swAu9hVO1w=="}' 2024-03-25T23:34:35.848 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 614c4a39-b13c-4192-ba13-0218e9f2bcd7 -i td/osd-scrub-repair/3/new.json 2024-03-25T23:34:36.238 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:34:36.255 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-03-25T23:34:36.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQALCgJmeEC4MhAA5aZ6GPLQFpr7swAu9hVO1w== --osd-uuid 614c4a39-b13c-4192-ba13-0218e9f2bcd7 2024-03-25T23:34:36.290 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:34:36.289+0000 7f7a125e17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:34:36.340 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:34:36.341+0000 7f7a125e17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:34:36.342 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:34:36.341+0000 7f7a125e17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:34:36.342 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:34:36.345+0000 7f7a125e17c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-25T23:34:36.342 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:34:36.345+0000 7f7a125e17c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-25T23:34:36.342 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:34:36.345+0000 7f7a125e17c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-25T23:34:36.425 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:34:36.425+0000 7f7a125e17c0 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-03-25T23:34:42.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-03-25T23:34:42.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:34:42.869 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd3 key to auth repository 2024-03-25T23:34:42.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-03-25T23:34:42.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:34:43.252 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-03-25T23:34:43.253 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.3 2024-03-25T23:34:43.253 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:34:43.253 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:34:43.253 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:34:43.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:34:43.285 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:34:43.285+0000 7fafb89c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:34:43.287 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:34:43.289+0000 7fafb89c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:34:43.289 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:34:43.289+0000 7fafb89c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:34:43.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-03-25T23:34:43.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:34:43.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-25T23:34:43.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:34:43.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:34:43.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:34:43.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:34:43.599 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:34:43.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:34:43.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:34:43.869 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:34:43.869+0000 7fafb89c97c0 -1 Falling back to public interface 2024-03-25T23:34:43.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:34:44.921 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:34:44.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:34:44.922 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:34:44.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:34:44.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:34:44.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:34:45.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:34:46.244 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:34:46.244 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:34:46.245 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:34:46.245 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:34:46.245 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:34:46.245 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:34:46.559 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:34:46.561+0000 7fafb89c97c0 -1 osd.3 0 log_to_monitors true 2024-03-25T23:34:46.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:34:47.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:34:47.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:34:47.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:34:47.588 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:34:47.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:34:47.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:34:47.996 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:34:48.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:34:48.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:34:48.998 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:34:48.999 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:34:48.999 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:34:48.999 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:34:49.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:34:50.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:34:50.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:34:50.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:34:50.345 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:34:50.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:34:50.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:34:50.791 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:34:51.792 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:34:51.792 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:34:51.793 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:34:51.793 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:34:51.793 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:34:51.793 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:34:52.115 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:34:53.117 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:34:53.117 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:34:53.117 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:34:53.118 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:34:53.118 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:34:53.118 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:34:53.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:34:54.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:34:54.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:34:54.435 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:34:54.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:34:54.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:34:54.436 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:34:54.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:34:55.975 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:34:55.975 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:34:55.975 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:34:55.976 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:34:55.976 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:34:55.976 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:34:56.285 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:34:57.286 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:34:57.286 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:34:57.287 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:34:57.287 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:34:57.287 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:34:57.287 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:34:57.605 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:34:58.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:34:58.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:34:58.607 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:34:58.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:34:58.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:34:58.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:34:59.101 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:35:00.103 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:35:00.103 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:35:00.103 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:35:00.104 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:35:00.104 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:35:00.104 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:35:00.416 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:35:01.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:35:01.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:35:01.418 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:35:01.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:35:01.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:35:01.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:35:02.035 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:35:03.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:35:03.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:35:03.037 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:35:03.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:35:03.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:35:03.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:35:03.385 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:35:04.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:35:04.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:35:04.387 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:35:04.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:35:04.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:35:04.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:35:05.206 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:35:06.208 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:35:06.208 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:35:06.208 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:35:06.209 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:35:06.209 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:35:06.209 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:35:07.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:35:08.170 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:35:08.170 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:35:08.170 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:35:08.171 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:35:08.171 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:35:08.171 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:35:08.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:35:09.968 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:35:09.968 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:35:09.968 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:35:09.968 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:35:09.969 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:35:09.969 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:35:10.674 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:35:11.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:35:11.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:35:11.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:35:11.677 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:35:11.677 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:35:11.677 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:35:12.185 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:35:13.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:35:13.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:35:13.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-25T23:35:13.188 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-25T23:35:13.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:35:13.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:35:13.507 INFO:tasks.workunit.client.0.smithi129.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2429907766,v1:127.0.0.1:6827/2429907766] [v2:127.0.0.1:6828/2429907766,v1:127.0.0.1:6829/2429907766] exists,up 614c4a39-b13c-4192-ba13-0218e9f2bcd7 2024-03-25T23:35:13.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:35:13.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:35:13.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:35:13.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:807: corrupt_and_repair_jerasure: create_rbd_pool 2024-03-25T23:35:13.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-25T23:35:13.807 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'rbd' does not exist 2024-03-25T23:35:13.823 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-25T23:35:13.823 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-25T23:35:14.401 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'rbd' created 2024-03-25T23:35:14.423 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T23:35:15.425 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-25T23:35:17.954 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: corrupt_and_repair_jerasure: wait_for_clean 2024-03-25T23:35:17.954 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:35:17.954 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:35:17.954 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:35:17.954 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:35:17.955 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:35:17.955 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:35:17.956 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:35:17.956 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:35:17.956 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:35:18.082 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:35:18.082 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:35:18.082 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:35:18.083 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:35:18.083 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:35:18.083 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:35:18.403 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:35:18.403 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:35:18.403 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:35:18.403 INFO:tasks.workunit.client.0.smithi129.stderr:3' 2024-03-25T23:35:18.403 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:35:18.404 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:35:18.404 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:35:18.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836508 2024-03-25T23:35:18.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836508 2024-03-25T23:35:18.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508' 2024-03-25T23:35:18.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:35:18.553 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:35:18.696 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672980 2024-03-25T23:35:18.696 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672980 2024-03-25T23:35:18.696 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672980' 2024-03-25T23:35:18.696 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:35:18.696 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:35:18.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509452 2024-03-25T23:35:18.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509452 2024-03-25T23:35:18.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672980 2-64424509452' 2024-03-25T23:35:18.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:35:18.836 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:35:18.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345930 2024-03-25T23:35:18.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345930 2024-03-25T23:35:18.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672980 2-64424509452 3-85899345930' 2024-03-25T23:35:18.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:35:18.980 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:35:18.980 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836508 2024-03-25T23:35:18.980 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:35:18.981 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836508 2024-03-25T23:35:18.981 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:35:18.982 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836508 2024-03-25T23:35:18.982 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 21474836508 2024-03-25T23:35:18.983 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-03-25T23:35:18.983 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:35:19.282 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836507 -lt 21474836508 2024-03-25T23:35:19.282 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:35:20.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:35:20.284 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:35:20.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836508 2024-03-25T23:35:20.635 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:35:20.635 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672980 2024-03-25T23:35:20.635 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:35:20.636 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:35:20.637 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672980 2024-03-25T23:35:20.638 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:35:20.640 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672980 2024-03-25T23:35:20.640 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-03-25T23:35:20.640 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 42949672980 2024-03-25T23:35:20.640 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:35:21.014 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672980 -lt 42949672980 2024-03-25T23:35:21.014 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:35:21.014 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509452 2024-03-25T23:35:21.015 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:35:21.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:35:21.018 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:35:21.018 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509452 2024-03-25T23:35:21.019 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509452 2024-03-25T23:35:21.019 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 64424509452 2024-03-25T23:35:21.020 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-03-25T23:35:21.020 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:35:21.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509452 -lt 64424509452 2024-03-25T23:35:21.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:35:21.330 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:35:21.330 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345930 2024-03-25T23:35:21.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:35:21.331 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345930 2024-03-25T23:35:21.331 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:35:21.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345930 2024-03-25T23:35:21.333 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.3 seq 85899345930 2024-03-25T23:35:21.333 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345930' 2024-03-25T23:35:21.333 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:35:21.629 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345930 -lt 85899345930 2024-03-25T23:35:21.629 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:35:21.630 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:35:21.630 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:35:22.115 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-25T23:35:22.115 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:35:22.115 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:35:22.115 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:35:22.116 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:35:22.116 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:35:22.116 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:35:22.116 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:35:22.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T23:35:22.416 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:35:22.417 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:35:22.417 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:35:22.764 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-25T23:35:22.764 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:35:22.764 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:35:22.764 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:810: corrupt_and_repair_jerasure: create_ec_pool ecpool false k=2 m=2 2024-03-25T23:35:22.764 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-25T23:35:22.764 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-25T23:35:22.764 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-03-25T23:35:22.764 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-25T23:35:22.764 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-03-25T23:35:23.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-25T23:35:23.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-25T23:35:24.005 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'ecpool' created 2024-03-25T23:35:24.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T23:35:25.029 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-03-25T23:35:25.029 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-25T23:35:25.029 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:35:25.029 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:35:25.029 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:35:25.030 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:35:25.030 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:35:25.031 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:35:25.031 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:35:25.031 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:35:25.031 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:35:25.158 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:35:25.158 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:35:25.158 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:35:25.159 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:35:25.159 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:35:25.159 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:35:25.534 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:35:25.534 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:35:25.534 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:35:25.534 INFO:tasks.workunit.client.0.smithi129.stderr:3' 2024-03-25T23:35:25.534 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:35:25.534 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:35:25.534 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:35:25.709 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836510 2024-03-25T23:35:25.709 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836510 2024-03-25T23:35:25.709 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510' 2024-03-25T23:35:25.709 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:35:25.710 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:35:25.857 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672982 2024-03-25T23:35:25.857 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672982 2024-03-25T23:35:25.857 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510 1-42949672982' 2024-03-25T23:35:25.857 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:35:25.857 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:35:26.028 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509455 2024-03-25T23:35:26.029 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509455 2024-03-25T23:35:26.029 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510 1-42949672982 2-64424509455' 2024-03-25T23:35:26.029 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:35:26.029 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:35:26.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345932 2024-03-25T23:35:26.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345932 2024-03-25T23:35:26.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510 1-42949672982 2-64424509455 3-85899345932' 2024-03-25T23:35:26.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:35:26.173 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836510 2024-03-25T23:35:26.173 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:35:26.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:35:26.174 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836510 2024-03-25T23:35:26.174 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:35:26.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836510 2024-03-25T23:35:26.175 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 21474836510 2024-03-25T23:35:26.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-03-25T23:35:26.176 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:35:26.521 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836510 -lt 21474836510 2024-03-25T23:35:26.521 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:35:26.522 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672982 2024-03-25T23:35:26.522 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:35:26.523 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:35:26.524 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672982 2024-03-25T23:35:26.524 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:35:26.525 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672982 2024-03-25T23:35:26.526 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 42949672982 2024-03-25T23:35:26.526 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-03-25T23:35:26.526 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:35:26.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672981 -lt 42949672982 2024-03-25T23:35:26.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:35:27.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:35:27.840 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:35:28.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672983 -lt 42949672982 2024-03-25T23:35:28.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:35:28.143 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:35:28.143 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509455 2024-03-25T23:35:28.144 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:35:28.145 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509455 2024-03-25T23:35:28.145 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:35:28.146 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509455 2024-03-25T23:35:28.146 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 64424509455 2024-03-25T23:35:28.147 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-03-25T23:35:28.147 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:35:28.475 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509455 -lt 64424509455 2024-03-25T23:35:28.475 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:35:28.476 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345932 2024-03-25T23:35:28.476 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:35:28.478 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:35:28.479 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345932 2024-03-25T23:35:28.479 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:35:28.482 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345932 2024-03-25T23:35:28.482 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.3 seq 85899345932 2024-03-25T23:35:28.483 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345932' 2024-03-25T23:35:28.483 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:35:28.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345932 -lt 85899345932 2024-03-25T23:35:28.790 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:35:28.790 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:35:28.790 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:35:29.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:35:29.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:35:29.195 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:35:29.195 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:35:29.195 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:35:29.195 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:35:29.195 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:35:29.195 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:35:29.531 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:35:29.532 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:35:29.532 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:35:29.532 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:35:29.909 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:35:29.909 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:35:29.909 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:35:29.909 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-25T23:35:29.909 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:811: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-03-25T23:35:29.909 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-03-25T23:35:29.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-03-25T23:35:29.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-03-25T23:35:29.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-25T23:35:29.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-25T23:35:29.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-03-25T23:35:29.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-25T23:35:29.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-25T23:35:29.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-25T23:35:30.373 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-25T23:35:30.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-25T23:35:30.800 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-25T23:35:30.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-25T23:35:30.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-25T23:35:30.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-25T23:35:31.267 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-03-25T23:35:31.267 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-03-25T23:35:31.267 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-25T23:35:31.267 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-03-25T23:35:31.268 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-25T23:35:31.611 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-03-25T23:35:31.612 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-03-25T23:35:31.612 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-03-25T23:35:31.612 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-25T23:35:31.612 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-03-25T23:35:31.613 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-03-25T23:35:31.613 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T23:35:31.928 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-25T23:35:31.928 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:35:31.928 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:35:31.929 INFO:tasks.workunit.client.0.smithi129.stderr:0' 2024-03-25T23:35:31.929 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 0 2024-03-25T23:35:31.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('1' '2' '0') 2024-03-25T23:35:31.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-03-25T23:35:31.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=1 2024-03-25T23:35:31.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=2 2024-03-25T23:35:31.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-03-25T23:35:31.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-25T23:35:31.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-03-25T23:35:31.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-03-25T23:35:31.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-25T23:35:31.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:35:31.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:35:31.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-25T23:35:31.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:35:31.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-03-25T23:35:31.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:35:31.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:35:31.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-25T23:35:31.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:35:31.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-25T23:35:31.930 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:35:31.930 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:35:31.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:35:31.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:35:31.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:35:32.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:35:32.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-03-25T23:35:32.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:35:32.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:35:32.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-25T23:35:32.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:35:32.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-25T23:35:32.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-03-25T23:35:33.942 INFO:tasks.workunit.client.0.smithi129.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-03-25T23:35:34.506 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-25T23:35:34.506 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:35:34.506 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:35:34.506 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-25T23:35:34.506 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:35:34.506 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-25T23:35:34.506 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:35:34.506 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:35:34.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:35:34.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:35:34.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-25T23:35:34.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-25T23:35:34.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:35:34.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:35:34.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:35:34.507 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:35:34.507 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:35:34.507 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:35:34.508 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:35:34.508 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:35:34.508 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:35:34.509 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:35:34.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:35:34.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:35:34.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:35:34.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:35:34.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:35:34.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:35:34.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:35:34.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:35:34.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:35:34.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:35:34.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-25T23:35:34.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-25T23:35:34.510 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.3 2024-03-25T23:35:34.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:35:34.510 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-25T23:35:34.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-25T23:35:34.512 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:35:34.512 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:35:34.512 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:35:34.543 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:35:34.544+0000 7f158f38f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:35:34.546 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:35:34.548+0000 7f158f38f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:35:34.548 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:35:34.548+0000 7f158f38f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:35:34.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-25T23:35:34.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:35:34.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-25T23:35:34.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:35:34.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:35:34.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:35:34.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:35:34.835 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-25T23:35:34.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:35:34.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:35:35.089 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:35:35.088+0000 7f158f38f7c0 -1 Falling back to public interface 2024-03-25T23:35:35.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:35:36.155 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:35:36.155 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:35:36.155 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:35:36.155 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:35:36.156 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:35:36.156 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:35:36.469 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:35:37.471 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:35:37.471 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:35:37.471 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:35:37.471 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:35:37.471 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:35:37.471 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:35:37.641 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:35:37.640+0000 7f158f38f7c0 -1 osd.3 36 log_to_monitors true 2024-03-25T23:35:37.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:35:38.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:35:38.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:35:38.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:35:38.780 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:35:38.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:35:38.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:35:39.162 INFO:tasks.workunit.client.0.smithi129.stderr:osd.3 up in weight 1 up_from 40 up_thru 40 down_at 37 last_clean_interval [20,36) [v2:127.0.0.1:6826/1579482732,v1:127.0.0.1:6827/1579482732] [v2:127.0.0.1:6828/1579482732,v1:127.0.0.1:6829/1579482732] exists,up 614c4a39-b13c-4192-ba13-0218e9f2bcd7 2024-03-25T23:35:39.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:35:39.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:35:39.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:35:39.163 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:35:39.163 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:35:39.163 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:35:39.163 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:35:39.163 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:35:39.163 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:35:39.163 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:35:39.163 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:35:39.163 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:35:39.163 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:35:39.271 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:35:39.271 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:35:39.271 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:35:39.271 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:35:39.272 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:35:39.272 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:35:39.636 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:35:39.637 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:35:39.637 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:35:39.637 INFO:tasks.workunit.client.0.smithi129.stderr:3' 2024-03-25T23:35:39.637 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:35:39.637 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:35:39.637 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:35:39.906 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836514 2024-03-25T23:35:39.906 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836514 2024-03-25T23:35:39.906 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514' 2024-03-25T23:35:39.906 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:35:39.906 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:35:40.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672986 2024-03-25T23:35:40.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672986 2024-03-25T23:35:40.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514 1-42949672986' 2024-03-25T23:35:40.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:35:40.254 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:35:40.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509458 2024-03-25T23:35:40.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509458 2024-03-25T23:35:40.432 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514 1-42949672986 2-64424509458' 2024-03-25T23:35:40.432 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:35:40.432 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:35:40.575 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-03-25T23:35:40.575 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-03-25T23:35:40.575 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514 1-42949672986 2-64424509458 3-171798691842' 2024-03-25T23:35:40.575 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:35:40.575 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836514 2024-03-25T23:35:40.575 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:35:40.576 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:35:40.576 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836514 2024-03-25T23:35:40.576 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:35:40.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836514 2024-03-25T23:35:40.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836514' 2024-03-25T23:35:40.577 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 21474836514 2024-03-25T23:35:40.578 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:35:40.886 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836513 -lt 21474836514 2024-03-25T23:35:40.886 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:35:41.888 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:35:41.889 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:35:42.215 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836515 -lt 21474836514 2024-03-25T23:35:42.215 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:35:42.215 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672986 2024-03-25T23:35:42.215 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:35:42.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:35:42.218 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672986 2024-03-25T23:35:42.218 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:35:42.220 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672986 2024-03-25T23:35:42.220 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2024-03-25T23:35:42.220 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 42949672986 2024-03-25T23:35:42.220 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:35:42.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672986 -lt 42949672986 2024-03-25T23:35:42.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:35:42.561 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509458 2024-03-25T23:35:42.562 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:35:42.563 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:35:42.564 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509458 2024-03-25T23:35:42.564 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:35:42.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509458 2024-03-25T23:35:42.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-03-25T23:35:42.565 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 64424509458 2024-03-25T23:35:42.566 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:35:42.904 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509459 -lt 64424509458 2024-03-25T23:35:42.904 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:35:42.904 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-171798691842 2024-03-25T23:35:42.905 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:35:42.906 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:35:42.907 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-171798691842 2024-03-25T23:35:42.907 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:35:42.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-03-25T23:35:42.909 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 171798691842' 2024-03-25T23:35:42.909 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.3 seq 171798691842 2024-03-25T23:35:42.909 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:35:43.218 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691842 -lt 171798691842 2024-03-25T23:35:43.219 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:35:43.219 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:35:43.219 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:35:43.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:35:43.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:35:43.615 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:35:43.615 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:35:43.615 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:35:43.615 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:35:43.616 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:35:43.616 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:35:43.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:35:43.922 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:35:43.923 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:35:43.923 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:35:44.285 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:35:44.285 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:35:44.285 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:35:44.286 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-03-25T23:35:44.286 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-25T23:35:44.286 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-25T23:35:44.286 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-25T23:35:44.286 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-25T23:35:44.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-03-25T23:35:44.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-03-25T23:35:44.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-25T23:35:44.610 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-25T23:35:44.610 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:35:44.610 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:35:44.610 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:35:44.610 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:35:44.947 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-25T23:35:23.915767+0000 2024-03-25T23:35:44.947 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-25T23:35:45.222 INFO:tasks.workunit.client.0.smithi129.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-25T23:35:45.239 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-25T23:35:23.915767+0000 2024-03-25T23:35:45.239 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-25T23:35:45.239 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-25T23:35:23.915767+0000 2024-03-25T23:35:45.239 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-25T23:35:45.239 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-25T23:35:45.239 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:35:45.239 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:35:45.239 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:35:45.239 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:35:45.239 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:35:45.239 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:35:45.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:35:23.915767+0000 '>' 2024-03-25T23:35:23.915767+0000 2024-03-25T23:35:45.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:35:46.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:35:46.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:35:46.569 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:35:46.569 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:35:46.569 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:35:46.570 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:35:46.570 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:35:46.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:35:23.915767+0000 '>' 2024-03-25T23:35:23.915767+0000 2024-03-25T23:35:46.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:35:47.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:35:47.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:35:47.923 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:35:47.923 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:35:47.923 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:35:47.923 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:35:47.923 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:35:48.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:35:23.915767+0000 '>' 2024-03-25T23:35:23.915767+0000 2024-03-25T23:35:48.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:35:49.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:35:49.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:35:49.239 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:35:49.239 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:35:49.239 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:35:49.240 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:35:49.240 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:35:49.552 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:35:23.915767+0000 '>' 2024-03-25T23:35:23.915767+0000 2024-03-25T23:35:49.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:35:50.554 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:35:50.554 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:35:50.555 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:35:50.555 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:35:50.555 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:35:50.555 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:35:50.556 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:35:50.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:35:45.684162+0000 '>' 2024-03-25T23:35:23.915767+0000 2024-03-25T23:35:50.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-25T23:35:50.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-25T23:35:50.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:35:50.875 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:35:50.875 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-25T23:35:50.875 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:35:50.875 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-25T23:35:50.875 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:35:50.875 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:35:50.875 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-25T23:35:50.875 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:35:50.875 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-25T23:35:50.875 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:35:50.875 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:35:50.875 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:35:50.875 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:35:50.875 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:35:51.184 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:35:51.184 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-25T23:35:51.184 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:35:51.184 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:35:51.184 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-25T23:35:51.184 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:35:51.184 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-25T23:35:51.184 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-03-25T23:35:51.419 INFO:tasks.workunit.client.0.smithi129.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-03-25T23:35:51.419 INFO:tasks.workunit.client.0.smithi129.stdout:_ 2024-03-25T23:35:51.419 INFO:tasks.workunit.client.0.smithi129.stdout:hinfo_key 2024-03-25T23:35:51.419 INFO:tasks.workunit.client.0.smithi129.stdout:snapset 2024-03-25T23:35:51.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-25T23:35:51.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:35:51.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:35:51.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-25T23:35:51.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:35:51.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-25T23:35:51.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:35:51.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:35:51.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:35:51.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:35:51.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-25T23:35:51.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-25T23:35:51.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:35:51.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:35:51.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:35:51.707 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:35:51.707 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:35:51.707 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:35:51.709 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:35:51.709 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:35:51.709 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:35:51.709 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:35:51.709 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:35:51.709 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:35:51.709 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:35:51.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:35:51.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:35:51.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:35:51.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:35:51.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:35:51.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:35:51.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:35:51.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-25T23:35:51.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-25T23:35:51.712 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.3 2024-03-25T23:35:51.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:35:51.713 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-25T23:35:51.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-25T23:35:51.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:35:51.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:35:51.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:35:51.744 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:35:51.744+0000 7f4aa92b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:35:51.747 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:35:51.748+0000 7f4aa92b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:35:51.748 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:35:51.748+0000 7f4aa92b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:35:52.072 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-25T23:35:52.072 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:35:52.073 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-25T23:35:52.073 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:35:52.073 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:35:52.073 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:35:52.073 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:35:52.073 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-25T23:35:52.073 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:35:52.073 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:35:52.292 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:35:52.292+0000 7f4aa92b97c0 -1 Falling back to public interface 2024-03-25T23:35:52.524 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:35:53.525 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:35:53.525 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:35:53.526 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:35:53.526 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:35:53.526 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:35:53.526 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:35:53.846 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:35:54.847 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:35:54.847 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:35:54.847 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:35:54.847 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:35:54.847 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:35:54.847 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:35:55.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:35:55.289 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:35:55.287+0000 7f4aa92b97c0 -1 osd.3 42 log_to_monitors true 2024-03-25T23:35:56.170 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:35:56.170 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:35:56.170 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:35:56.170 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:35:56.171 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:35:56.171 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:35:56.549 INFO:tasks.workunit.client.0.smithi129.stderr:osd.3 up in weight 1 up_from 46 up_thru 46 down_at 43 last_clean_interval [40,42) [v2:127.0.0.1:6826/2132588965,v1:127.0.0.1:6827/2132588965] [v2:127.0.0.1:6828/2132588965,v1:127.0.0.1:6829/2132588965] exists,up 614c4a39-b13c-4192-ba13-0218e9f2bcd7 2024-03-25T23:35:56.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:35:56.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:35:56.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:35:56.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:35:56.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:35:56.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:35:56.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:35:56.551 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:35:56.551 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:35:56.551 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:35:56.551 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:35:56.551 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:35:56.551 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:35:56.652 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:35:56.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:35:56.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:35:56.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:35:56.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:35:56.653 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:35:57.155 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:35:57.155 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:35:57.155 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:35:57.155 INFO:tasks.workunit.client.0.smithi129.stderr:3' 2024-03-25T23:35:57.155 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:35:57.155 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:35:57.156 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:35:57.305 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836519 2024-03-25T23:35:57.305 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836519 2024-03-25T23:35:57.305 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836519' 2024-03-25T23:35:57.305 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:35:57.305 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:35:57.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672991 2024-03-25T23:35:57.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672991 2024-03-25T23:35:57.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836519 1-42949672991' 2024-03-25T23:35:57.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:35:57.464 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:35:57.606 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509463 2024-03-25T23:35:57.606 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509463 2024-03-25T23:35:57.606 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836519 1-42949672991 2-64424509463' 2024-03-25T23:35:57.606 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:35:57.606 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:35:57.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=197568495618 2024-03-25T23:35:57.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 197568495618 2024-03-25T23:35:57.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836519 1-42949672991 2-64424509463 3-197568495618' 2024-03-25T23:35:57.747 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:35:57.747 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836519 2024-03-25T23:35:57.747 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:35:57.748 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:35:57.748 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836519 2024-03-25T23:35:57.748 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:35:57.749 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836519 2024-03-25T23:35:57.749 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836519' 2024-03-25T23:35:57.749 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 21474836519 2024-03-25T23:35:57.750 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:35:58.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836519 -lt 21474836519 2024-03-25T23:35:58.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:35:58.038 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672991 2024-03-25T23:35:58.039 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:35:58.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:35:58.041 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:35:58.041 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672991 2024-03-25T23:35:58.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672991 2024-03-25T23:35:58.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672991' 2024-03-25T23:35:58.042 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 42949672991 2024-03-25T23:35:58.042 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:35:58.360 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672991 -lt 42949672991 2024-03-25T23:35:58.361 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:35:58.361 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:35:58.362 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509463 2024-03-25T23:35:58.362 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:35:58.363 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509463 2024-03-25T23:35:58.363 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:35:58.365 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509463 2024-03-25T23:35:58.365 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2024-03-25T23:35:58.365 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 64424509463 2024-03-25T23:35:58.365 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:35:58.679 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509463 -lt 64424509463 2024-03-25T23:35:58.679 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:35:58.680 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-197568495618 2024-03-25T23:35:58.681 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:35:58.682 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:35:58.683 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-197568495618 2024-03-25T23:35:58.683 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:35:58.685 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=197568495618 2024-03-25T23:35:58.685 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 197568495618' 2024-03-25T23:35:58.685 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.3 seq 197568495618 2024-03-25T23:35:58.685 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:35:59.009 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 197568495618 -lt 197568495618 2024-03-25T23:35:59.009 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:35:59.010 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:35:59.010 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:35:59.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:35:59.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:35:59.392 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:35:59.392 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:35:59.392 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:35:59.393 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:35:59.393 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:35:59.393 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:35:59.717 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T23:35:59.718 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:35:59.718 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:35:59.718 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:36:00.077 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-25T23:36:00.077 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-25T23:36:00.077 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-25T23:36:00.077 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-25T23:36:00.077 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:36:00.077 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-25T23:36:00.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:36:00.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:36:00.179 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:36:00.179 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:36:00.179 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:36:00.179 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:36:00.180 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:36:00.180 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:36:00.473 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T23:36:00.474 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:36:00.474 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:36:00.474 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:36:00.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-25T23:36:00.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-25T23:36:00.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-25T23:36:00.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-25T23:36:00.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-25T23:36:00.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-25T23:36:00.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-25T23:36:00.835 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-25T23:36:00.835 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-25T23:36:01.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-25T23:36:01.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-25T23:36:01.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-25T23:36:01.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:36:01.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-25T23:36:01.437 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:36:01.437 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:36:01.438 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:36:01.438 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:36:01.438 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:36:01.438 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:36:01.439 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:36:01.439 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:36:01.737 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T23:36:01.738 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:36:01.738 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:36:01.738 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:36:02.117 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-25T23:36:02.118 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-25T23:36:02.118 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-25T23:36:02.118 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-25T23:36:02.118 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-25T23:36:02.118 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-25T23:36:02.118 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-25T23:36:02.118 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-25T23:36:02.119 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-25T23:36:02.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-25T23:36:02.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-25T23:36:02.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-25T23:36:02.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:36:02.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-25T23:36:02.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:36:02.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:36:02.908 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:36:02.908 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:36:02.908 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:36:02.908 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:36:02.909 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:36:02.909 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:36:03.210 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:36:03.210 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:36:03.211 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:36:03.211 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:36:03.584 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:36:03.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:36:03.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:36:03.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-25T23:36:03.643 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-25T23:36:03.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2024-03-25T23:36:03.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-25T23:36:03.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-03-25T23:36:03.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-03-25T23:36:03.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-25T23:36:03.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:36:03.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:36:03.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-25T23:36:03.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:36:03.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-25T23:36:03.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:36:03.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:36:03.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-25T23:36:03.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:36:03.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-25T23:36:03.646 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:36:03.646 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:36:03.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:36:03.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:36:03.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:36:03.953 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:36:03.953 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-25T23:36:03.953 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:36:03.953 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:36:03.953 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-25T23:36:03.953 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:36:03.954 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:36:03.954 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-25T23:36:06.299 INFO:tasks.workunit.client.0.smithi129.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-03-25T23:36:06.837 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-25T23:36:06.837 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:36:06.837 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:36:06.837 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-25T23:36:06.837 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:36:06.837 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:36:06.837 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:36:06.837 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:36:06.837 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:36:06.837 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:36:06.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-25T23:36:06.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-25T23:36:06.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:36:06.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:36:06.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:36:06.838 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:36:06.838 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:36:06.838 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:36:06.839 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:36:06.839 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:36:06.839 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:36:06.839 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:36:06.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:36:06.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:36:06.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:36:06.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:36:06.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:36:06.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:36:06.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:36:06.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:36:06.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:36:06.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:36:06.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-25T23:36:06.841 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-25T23:36:06.841 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.1 2024-03-25T23:36:06.841 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:36:06.841 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-25T23:36:06.842 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-25T23:36:06.842 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:36:06.842 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:36:06.843 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:36:06.875 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:36:06.875+0000 7f0ecbdea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:36:06.877 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:36:06.875+0000 7f0ecbdea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:36:06.879 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:36:06.879+0000 7f0ecbdea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:36:07.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-25T23:36:07.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:36:07.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T23:36:07.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:36:07.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:36:07.158 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:36:07.158 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:36:07.158 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-25T23:36:07.158 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:36:07.158 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:36:07.413 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:36:07.411+0000 7f0ecbdea7c0 -1 Falling back to public interface 2024-03-25T23:36:07.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:36:08.470 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:36:08.470 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:36:08.470 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:36:08.470 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:36:08.470 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:36:08.470 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:36:08.786 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:36:09.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:36:09.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:36:09.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:36:09.789 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:36:09.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:36:09.790 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:36:10.110 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:36:10.283 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:36:10.283+0000 7f0ecbdea7c0 -1 osd.1 48 log_to_monitors true 2024-03-25T23:36:11.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:36:11.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:36:11.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:36:11.112 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:36:11.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:36:11.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:36:11.528 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1 up in weight 1 up_from 53 up_thru 53 down_at 49 last_clean_interval [10,48) [v2:127.0.0.1:6810/421162533,v1:127.0.0.1:6811/421162533] [v2:127.0.0.1:6812/421162533,v1:127.0.0.1:6813/421162533] exists,up 0a1d267b-774d-4fda-8cc0-7d86cd87d221 2024-03-25T23:36:11.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:36:11.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:36:11.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:36:11.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:36:11.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:36:11.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:36:11.529 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:36:11.529 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:36:11.529 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:36:11.530 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:36:11.530 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:36:11.530 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:36:11.530 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:36:11.637 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:36:11.637 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:36:11.637 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:36:11.637 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:36:11.637 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:36:11.637 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:36:12.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:36:12.173 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:36:12.173 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:36:12.173 INFO:tasks.workunit.client.0.smithi129.stderr:3' 2024-03-25T23:36:12.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:36:12.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:36:12.173 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:36:12.440 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836523 2024-03-25T23:36:12.440 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836523 2024-03-25T23:36:12.440 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836523' 2024-03-25T23:36:12.440 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:36:12.440 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:36:12.583 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=227633266690 2024-03-25T23:36:12.583 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 227633266690 2024-03-25T23:36:12.584 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836523 1-227633266690' 2024-03-25T23:36:12.584 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:36:12.584 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:36:12.725 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509467 2024-03-25T23:36:12.726 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509467 2024-03-25T23:36:12.726 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836523 1-227633266690 2-64424509467' 2024-03-25T23:36:12.726 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:36:12.726 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:36:12.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=197568495622 2024-03-25T23:36:12.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 197568495622 2024-03-25T23:36:12.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836523 1-227633266690 2-64424509467 3-197568495622' 2024-03-25T23:36:12.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:36:12.869 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836523 2024-03-25T23:36:12.869 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:36:12.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:36:12.870 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836523 2024-03-25T23:36:12.870 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:36:12.871 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836523 2024-03-25T23:36:12.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836523' 2024-03-25T23:36:12.872 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 21474836523 2024-03-25T23:36:12.872 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:36:13.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836522 -lt 21474836523 2024-03-25T23:36:13.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:36:14.174 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:36:14.175 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:36:14.486 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836523 -lt 21474836523 2024-03-25T23:36:14.486 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:36:14.487 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-227633266690 2024-03-25T23:36:14.487 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:36:14.488 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:36:14.489 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:36:14.489 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-227633266690 2024-03-25T23:36:14.490 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=227633266690 2024-03-25T23:36:14.490 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 227633266690' 2024-03-25T23:36:14.490 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 227633266690 2024-03-25T23:36:14.490 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:36:14.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 227633266690 -lt 227633266690 2024-03-25T23:36:14.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:36:14.806 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509467 2024-03-25T23:36:14.806 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:36:14.807 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:36:14.808 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509467 2024-03-25T23:36:14.808 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:36:14.809 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509467 2024-03-25T23:36:14.809 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509467' 2024-03-25T23:36:14.809 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 64424509467 2024-03-25T23:36:14.810 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:36:15.146 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509467 -lt 64424509467 2024-03-25T23:36:15.147 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:36:15.150 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-197568495622 2024-03-25T23:36:15.150 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:36:15.150 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:36:15.150 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-197568495622 2024-03-25T23:36:15.150 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:36:15.151 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=197568495622 2024-03-25T23:36:15.151 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 197568495622' 2024-03-25T23:36:15.151 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.3 seq 197568495622 2024-03-25T23:36:15.151 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:36:15.482 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 197568495622 -lt 197568495622 2024-03-25T23:36:15.483 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:36:15.483 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:36:15.483 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:36:15.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:36:15.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:36:15.874 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:36:15.874 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:36:15.874 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:36:15.874 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:36:15.874 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:36:15.875 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:36:16.184 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T23:36:16.184 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:36:16.185 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:36:16.185 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:36:16.559 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-25T23:36:16.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-25T23:36:16.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-25T23:36:16.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-25T23:36:16.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:36:16.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-25T23:36:16.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:36:16.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:36:16.662 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:36:16.662 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:36:16.662 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:36:16.662 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:36:16.663 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:36:16.663 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:36:16.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T23:36:16.962 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:36:16.963 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:36:16.963 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:36:17.325 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-25T23:36:17.325 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-25T23:36:17.325 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-25T23:36:17.325 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-25T23:36:17.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-25T23:36:17.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-25T23:36:17.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-25T23:36:17.326 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-25T23:36:17.326 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-25T23:36:17.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-25T23:36:17.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-25T23:36:17.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-25T23:36:17.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:36:17.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-25T23:36:17.912 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:36:17.912 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:36:17.913 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:36:17.913 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:36:17.913 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:36:17.913 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:36:17.914 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:36:17.914 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:36:18.221 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:36:18.222 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:36:18.222 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:36:18.223 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:36:18.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:36:18.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:36:18.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:36:18.581 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-03-25T23:36:18.581 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-25T23:36:18.581 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-25T23:36:18.581 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-25T23:36:18.582 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-25T23:36:18.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-03-25T23:36:18.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-03-25T23:36:18.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-25T23:36:18.914 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-25T23:36:18.914 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:36:18.914 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:36:18.915 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:36:18.915 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:36:19.251 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-25T23:35:45.684162+0000 2024-03-25T23:36:19.251 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-25T23:36:19.547 INFO:tasks.workunit.client.0.smithi129.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-25T23:36:19.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-25T23:35:45.684162+0000 2024-03-25T23:36:19.563 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-25T23:36:19.563 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-25T23:35:45.684162+0000 2024-03-25T23:36:19.563 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-25T23:36:19.563 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-25T23:36:19.563 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:36:19.564 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:36:19.564 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:36:19.564 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:36:19.564 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:36:19.564 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:36:19.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:35:45.684162+0000 '>' 2024-03-25T23:35:45.684162+0000 2024-03-25T23:36:19.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:36:20.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:36:20.862 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:36:20.863 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:36:20.863 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:36:20.863 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:36:20.863 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:36:20.863 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:36:21.167 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:35:45.684162+0000 '>' 2024-03-25T23:35:45.684162+0000 2024-03-25T23:36:21.167 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:36:22.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:36:22.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:36:22.171 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:36:22.171 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:36:22.171 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:36:22.171 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:36:22.171 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:36:22.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:36:20.037815+0000 '>' 2024-03-25T23:35:45.684162+0000 2024-03-25T23:36:22.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-25T23:36:22.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-25T23:36:22.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:36:22.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:36:22.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-25T23:36:22.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:36:22.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-25T23:36:22.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:36:22.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:36:22.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-25T23:36:22.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:36:22.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-25T23:36:22.485 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:36:22.485 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:36:22.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:36:22.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:36:22.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:36:22.793 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:36:22.793 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-25T23:36:22.794 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:36:22.794 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:36:22.794 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-25T23:36:22.794 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:36:22.794 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-25T23:36:22.794 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-03-25T23:36:23.098 INFO:tasks.workunit.client.0.smithi129.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-03-25T23:36:23.098 INFO:tasks.workunit.client.0.smithi129.stdout:_ 2024-03-25T23:36:23.098 INFO:tasks.workunit.client.0.smithi129.stdout:hinfo_key 2024-03-25T23:36:23.098 INFO:tasks.workunit.client.0.smithi129.stdout:snapset 2024-03-25T23:36:23.395 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-25T23:36:23.395 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:36:23.395 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:36:23.395 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-25T23:36:23.395 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:36:23.395 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-25T23:36:23.396 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:36:23.396 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:36:23.396 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:36:23.396 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:36:23.396 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-25T23:36:23.396 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-25T23:36:23.396 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:36:23.396 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:36:23.396 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:36:23.396 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:36:23.396 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:36:23.396 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:36:23.396 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:36:23.396 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:36:23.396 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:36:23.397 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:36:23.398 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:36:23.398 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:36:23.398 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:36:23.398 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:36:23.399 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:36:23.399 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:36:23.399 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:36:23.399 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:36:23.399 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:36:23.399 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:36:23.399 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-25T23:36:23.399 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-25T23:36:23.399 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.3 2024-03-25T23:36:23.399 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:36:23.399 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-25T23:36:23.400 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-25T23:36:23.400 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:36:23.401 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:36:23.401 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:36:23.431 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:36:23.431+0000 7efd5adb07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:36:23.434 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:36:23.435+0000 7efd5adb07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:36:23.436 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:36:23.435+0000 7efd5adb07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:36:23.876 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-25T23:36:23.876 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:36:23.876 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-25T23:36:23.876 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:36:23.877 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:36:23.877 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:36:23.877 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:36:23.877 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-25T23:36:23.877 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:36:23.877 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:36:24.012 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:36:24.011+0000 7efd5adb07c0 -1 Falling back to public interface 2024-03-25T23:36:24.187 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:36:25.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:36:25.190 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:36:25.190 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:36:25.190 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:36:25.190 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:36:25.190 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:36:25.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:36:26.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:36:26.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:36:26.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:36:26.509 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:36:26.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:36:26.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:36:26.812 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:36:27.080 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:36:27.079+0000 7efd5adb07c0 -1 osd.3 55 log_to_monitors true 2024-03-25T23:36:27.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:36:27.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:36:27.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:36:27.814 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:36:27.815 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:36:27.815 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:36:28.302 INFO:tasks.workunit.client.0.smithi129.stderr:osd.3 up in weight 1 up_from 59 up_thru 59 down_at 56 last_clean_interval [46,55) [v2:127.0.0.1:6826/3086057932,v1:127.0.0.1:6827/3086057932] [v2:127.0.0.1:6828/3086057932,v1:127.0.0.1:6829/3086057932] exists,up 614c4a39-b13c-4192-ba13-0218e9f2bcd7 2024-03-25T23:36:28.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:36:28.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:36:28.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:36:28.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:36:28.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:36:28.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:36:28.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:36:28.303 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:36:28.304 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:36:28.304 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:36:28.304 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:36:28.304 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:36:28.304 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:36:28.408 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:36:28.408 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:36:28.408 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:36:28.408 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:36:28.408 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:36:28.408 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:36:28.894 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:36:28.894 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:36:28.894 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:36:28.894 INFO:tasks.workunit.client.0.smithi129.stderr:3' 2024-03-25T23:36:28.894 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:36:28.894 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:36:28.897 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:36:29.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836527 2024-03-25T23:36:29.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836527 2024-03-25T23:36:29.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527' 2024-03-25T23:36:29.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:36:29.046 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:36:29.187 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=227633266694 2024-03-25T23:36:29.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 227633266694 2024-03-25T23:36:29.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527 1-227633266694' 2024-03-25T23:36:29.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:36:29.188 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:36:29.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509471 2024-03-25T23:36:29.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509471 2024-03-25T23:36:29.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527 1-227633266694 2-64424509471' 2024-03-25T23:36:29.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:36:29.330 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:36:29.472 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070466 2024-03-25T23:36:29.472 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070466 2024-03-25T23:36:29.473 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527 1-227633266694 2-64424509471 3-253403070466' 2024-03-25T23:36:29.473 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:36:29.473 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836527 2024-03-25T23:36:29.473 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:36:29.474 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:36:29.475 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836527 2024-03-25T23:36:29.475 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:36:29.478 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836527 2024-03-25T23:36:29.478 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836527' 2024-03-25T23:36:29.478 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 21474836527 2024-03-25T23:36:29.478 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:36:29.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836526 -lt 21474836527 2024-03-25T23:36:29.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:36:30.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:36:30.785 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:36:31.107 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836527 -lt 21474836527 2024-03-25T23:36:31.107 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:36:31.107 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:36:31.107 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-227633266694 2024-03-25T23:36:31.108 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:36:31.109 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-227633266694 2024-03-25T23:36:31.109 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:36:31.110 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=227633266694 2024-03-25T23:36:31.110 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 227633266694' 2024-03-25T23:36:31.111 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 227633266694 2024-03-25T23:36:31.111 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:36:31.474 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 227633266694 -lt 227633266694 2024-03-25T23:36:31.474 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:36:31.475 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509471 2024-03-25T23:36:31.475 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:36:31.476 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:36:31.478 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509471 2024-03-25T23:36:31.478 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:36:31.478 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509471 2024-03-25T23:36:31.478 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509471' 2024-03-25T23:36:31.478 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 64424509471 2024-03-25T23:36:31.479 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:36:31.849 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509471 -lt 64424509471 2024-03-25T23:36:31.849 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:36:31.850 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070466 2024-03-25T23:36:31.851 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:36:31.853 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:36:31.854 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070466 2024-03-25T23:36:31.854 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:36:31.855 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070466 2024-03-25T23:36:31.855 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070466' 2024-03-25T23:36:31.855 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.3 seq 253403070466 2024-03-25T23:36:31.855 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:36:32.171 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070466 -lt 253403070466 2024-03-25T23:36:32.171 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:36:32.172 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:36:32.172 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:36:32.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:36:32.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:36:32.580 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:36:32.580 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:36:32.580 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:36:32.580 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:36:32.580 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:36:32.581 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:36:32.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:36:32.911 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:36:32.911 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:36:32.911 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:36:33.292 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:36:33.293 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:36:33.293 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:36:33.293 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-25T23:36:33.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-25T23:36:33.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2024-03-25T23:36:33.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-03-25T23:36:33.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-03-25T23:36:33.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=1 2024-03-25T23:36:33.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=2 2024-03-25T23:36:33.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-03-25T23:36:33.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-25T23:36:33.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:36:33.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:36:33.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 156876"' 2024-03-25T23:36:33.348 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 156876' 2024-03-25T23:36:33.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-03-25T23:36:33.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:36:33.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:36:33.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-25T23:36:33.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 156879"' 2024-03-25T23:36:33.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/156878: /' 2024-03-25T23:36:33.349 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 156879' 2024-03-25T23:36:33.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-03-25T23:36:33.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 156876 156879' 2024-03-25T23:36:33.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-25T23:36:33.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-25T23:36:33.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 156876 2024-03-25T23:36:33.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-03-25T23:36:33.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/156881: /' 2024-03-25T23:36:36.525 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:36:36.525 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:36:36.525 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-25T23:36:36.525 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:36:36.525 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-25T23:36:36.525 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:36:36.525 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:36:36.525 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-25T23:36:36.526 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:36:36.526 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-25T23:36:36.526 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:36:36.526 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:36:36.526 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:36:36.526 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:36:36.526 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:36:36.526 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:36:36.526 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-25T23:36:36.526 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:36:36.526 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:36:36.526 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-25T23:36:36.526 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:36:36.527 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:36:36.527 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-25T23:36:36.527 INFO:tasks.workunit.client.0.smithi129.stderr:156878: remove 1#2:eb822e21:::SOMETHING:head# 2024-03-25T23:36:36.527 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-25T23:36:36.527 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:36:36.527 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:36:36.527 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-25T23:36:36.527 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:36:36.527 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:36:36.527 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:36:36.527 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:36:36.530 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:36:36.530 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:36:36.530 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-25T23:36:36.530 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-25T23:36:36.531 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:36:36.531 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:36:36.531 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:36:36.531 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:36:36.531 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:36:36.531 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:36:36.531 INFO:tasks.workunit.client.0.smithi129.stderr:156878: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:36:36.531 INFO:tasks.workunit.client.0.smithi129.stderr:156878: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:36:36.531 INFO:tasks.workunit.client.0.smithi129.stderr:156878: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:36:36.531 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:36:36.531 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:36:36.531 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:36:36.531 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:36:36.531 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:36:36.531 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:36:36.532 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:36:36.532 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:36:36.532 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:36:36.532 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:36:36.532 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:36:36.532 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-25T23:36:36.532 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-25T23:36:36.532 INFO:tasks.workunit.client.0.smithi129.stderr:156878: start osd.1 2024-03-25T23:36:36.532 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:36:36.666 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:36:36.666 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:36:36.666 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-03-25T23:36:36.666 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:36:36.666 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-03-25T23:36:36.666 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:36:36.666 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:36:36.666 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-03-25T23:36:36.666 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:36:36.666 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-25T23:36:36.666 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:36:36.667 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:36:36.667 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:36:36.667 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:36:36.667 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:36:36.667 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:36:36.667 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-03-25T23:36:36.667 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:36:36.667 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:36:36.667 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-03-25T23:36:36.667 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:36:36.667 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-03-25T23:36:36.667 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-03-25T23:36:36.667 INFO:tasks.workunit.client.0.smithi129.stderr:156881: remove 2#2:eb822e21:::SOMETHING:head# 2024-03-25T23:36:36.667 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-03-25T23:36:36.668 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:36:36.668 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:36:36.668 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-25T23:36:36.668 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:36:36.668 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-25T23:36:36.668 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:36:36.668 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:36:36.669 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:36:36.669 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:36:36.669 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-25T23:36:36.669 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-25T23:36:36.669 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:36:36.669 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:36:36.669 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:36:36.669 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:36:36.669 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:36:36.669 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:36:36.669 INFO:tasks.workunit.client.0.smithi129.stderr:156881: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:36:36.669 INFO:tasks.workunit.client.0.smithi129.stderr:156881: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:36:36.669 INFO:tasks.workunit.client.0.smithi129.stderr:156881: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:36:36.669 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:36:36.670 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:36:36.670 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:36:36.670 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:36:36.670 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:36:36.670 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:36:36.670 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:36:36.670 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:36:36.670 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:36:36.670 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:36:36.670 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:36:36.670 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-25T23:36:36.670 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-25T23:36:36.670 INFO:tasks.workunit.client.0.smithi129.stderr:156881: start osd.2 2024-03-25T23:36:36.670 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:36:41.405 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-25T23:36:41.405 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-25T23:36:41.405 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:36:41.405 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:36:41.405 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:36:41.405 INFO:tasks.workunit.client.0.smithi129.stderr:156878: 2024-03-25T23:36:36.563+0000 7f327f5777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:36:41.405 INFO:tasks.workunit.client.0.smithi129.stderr:156878: 2024-03-25T23:36:36.567+0000 7f327f5777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:36:41.405 INFO:tasks.workunit.client.0.smithi129.stderr:156878: 2024-03-25T23:36:36.567+0000 7f327f5777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:36:41.405 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-25T23:36:41.406 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:36:41.406 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T23:36:41.406 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:36:41.406 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:36:41.406 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:36:41.406 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:36:41.406 INFO:tasks.workunit.client.0.smithi129.stderr:156878: 0 2024-03-25T23:36:41.406 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:36:41.406 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:36:41.406 INFO:tasks.workunit.client.0.smithi129.stderr:156878: 2024-03-25T23:36:37.131+0000 7f327f5777c0 -1 Falling back to public interface 2024-03-25T23:36:41.406 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:36:41.406 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:36:41.406 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:36:41.407 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:36:41.407 INFO:tasks.workunit.client.0.smithi129.stderr:156878: 1 2024-03-25T23:36:41.407 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:36:41.407 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:36:41.407 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:36:41.407 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:36:41.407 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:36:41.407 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:36:41.407 INFO:tasks.workunit.client.0.smithi129.stderr:156878: 2 2024-03-25T23:36:41.407 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:36:41.407 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:36:41.407 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:36:41.407 INFO:tasks.workunit.client.0.smithi129.stderr:156878: 2024-03-25T23:36:40.126+0000 7f327f5777c0 -1 osd.1 60 log_to_monitors true 2024-03-25T23:36:41.407 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:36:41.407 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:36:41.408 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:36:41.408 INFO:tasks.workunit.client.0.smithi129.stderr:156878: 3 2024-03-25T23:36:41.408 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:36:41.408 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:36:41.408 INFO:tasks.workunit.client.0.smithi129.stderr:156878: osd.1 up in weight 1 up_from 66 up_thru 56 down_at 61 last_156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-25T23:36:41.408 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-25T23:36:41.408 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:36:41.408 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:36:41.408 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:36:41.408 INFO:tasks.workunit.client.0.smithi129.stderr:156881: 2024-03-25T23:36:36.695+0000 7ff92ba617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:36:41.409 INFO:tasks.workunit.client.0.smithi129.stderr:156881: 2024-03-25T23:36:36.699+0000 7ff92ba617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:36:41.409 INFO:tasks.workunit.client.0.smithi129.stderr:156881: 2024-03-25T23:36:36.699+0000 7ff92ba617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:36:41.409 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-25T23:36:41.409 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:36:41.409 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-25T23:36:41.409 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:36:41.409 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:36:41.409 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:36:41.409 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:36:41.409 INFO:tasks.workunit.client.0.smithi129.stderr:156881: 0 2024-03-25T23:36:41.410 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:36:41.410 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:36:41.410 INFO:tasks.workunit.client.0.smithi129.stderr:156881: 2024-03-25T23:36:37.251+0000 7ff92ba617c0 -1 Falling back to public interface 2024-03-25T23:36:41.410 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:36:41.410 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:36:41.410 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:36:41.410 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:36:41.410 INFO:tasks.workunit.client.0.smithi129.stderr:156881: 1 2024-03-25T23:36:41.410 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:36:41.410 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:36:41.411 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:36:41.411 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:36:41.411 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:36:41.411 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:36:41.411 INFO:tasks.workunit.client.0.smithi129.stderr:156881: 2 2024-03-25T23:36:41.411 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:36:41.411 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:36:41.411 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:36:41.411 INFO:tasks.workunit.client.0.smithi129.stderr:156881: 2024-03-25T23:36:40.494+0000 7ff92ba617c0 -1 osd.2 60 log_to_monitors true 2024-03-25T23:36:41.411 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:36:41.411 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:36:41.411 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:36:41.412 INFO:tasks.workunit.client.0.smithi129.stderr:156881: 3 2024-03-25T23:36:41.412 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:36:41.412 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:36:42.553 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperclean_interval [53,60) [v2:127.0.0.1:6810/3146357908,v1:127.0.0.1:6811/3146357908] [v2:127.0.0.1:6812/3146357908,v1:127.0.0.1:6813/3146357908] exists,up 0a1d267b-774d-4fda-8cc0-7d86cd87d221 2024-03-25T23:36:42.553 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:36:42.553 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:36:42.553 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:36:42.554 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:36:42.554 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:36:42.554 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:36:42.554 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:36:42.554 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:36:42.554 INFO:tasks.workunit.client.0.smithi129.stderr:156878: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:36:42.555 INFO:tasks.workunit.client.0.smithi129.stderr:156878: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:36:42.555 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:36:42.555 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:36:42.555 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:36:42.555 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:36:42.555 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:36:42.555 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:36:42.555 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:36:42.555 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:36:42.555 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:36:42.555 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:36:42.555 INFO:tasks.workunit.client.0.smithi129.stderr:156878: 1 2024-03-25T23:36:42.555 INFO:tasks.workunit.client.0.smithi129.stderr:156878: 2 2024-03-25T23:36:42.556 INFO:tasks.workunit.client.0.smithi129.stderr:156878: 3' 2024-03-25T23:36:42.556 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:36:42.556 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:36:42.556 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:36:42.556 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836531 2024-03-25T23:36:42.556 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836531 2024-03-25T23:36:42.556 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531' 2024-03-25T23:36:42.556 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:36:42.556 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:36:42.556 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841538 2024-03-25T23:36:42.556 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841538 2024-03-25T23:36:42.556 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531 1-283467841538' 2024-03-25T23:36:42.557 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:36:43.530 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph s.sh:985: wait_for_osd: sleep 1 2024-03-25T23:36:43.530 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:36:43.530 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:36:43.530 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:36:43.530 INFO:tasks.workunit.client.0.smithi129.stderr:156881: 4 2024-03-25T23:36:43.530 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:36:43.530 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:36:43.530 INFO:tasks.workunit.client.0.smithi129.stderr:156881: osd.2 up in weight 1 up_from 67 up_thru 49 down_at 61 last_clean_interval [15,60) [v2:127.0.0.1:6818/3120384626,v1:127.0.0.1:6819/3120384626] [v2:127.0.0.1:6820/3120384626,v1:127.0.0.1:6821/3120384626] exists,up c816e86b-623d-482b-83dd-84132fd7b4ae 2024-03-25T23:36:43.530 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:36:43.531 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:36:43.531 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:36:43.531 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:36:43.531 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:36:43.531 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:36:43.531 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:36:43.531 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:36:43.531 INFO:tasks.workunit.client.0.smithi129.stderr:156881: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:36:43.531 INFO:tasks.workunit.client.0.smithi129.stderr:156881: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:36:43.531 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:36:43.531 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:36:43.531 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:36:43.531 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:36:43.531 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:36:43.532 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:36:43.532 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:36:43.532 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:36:43.532 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:36:43.532 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:36:43.532 INFO:tasks.workunit.client.0.smithi129.stderr:156881: 1 2024-03-25T23:36:43.532 INFO:tasks.workunit.client.0.smithi129.stderr:156881: 2 2024-03-25T23:36:43.532 INFO:tasks.workunit.client.0.smithi129.stderr:156881: 3' 2024-03-25T23:36:43.532 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:36:43.532 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:36:43.532 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:36:43.532 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836532 2024-03-25T23:36:43.532 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836532 2024-03-25T23:36:43.532 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836532' 2024-03-25T23:36:43.532 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:36:44.614 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:36:44.614 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841539 2024-03-25T23:36:44.615 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841539 2024-03-25T23:36:44.615 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836532 1-283467841539' 2024-03-25T23:36:44.615 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:36:44.615 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:36:44.615 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808835 2024-03-25T23:36:44.615 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808835 2024-03-25T23:36:44.616 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836532 1-283467841539 2-287762808835' 2024-03-25T23:36:44.616 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:36:44.616 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:36:44.616 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070471 2024-03-25T23:36:44.616 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070471 2024-03-25T23:36:44.616 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836532 1-283467841539 2-287762808835 3-253403070471' 2024-03-25T23:36:44.617 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:36:44.617 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836532 2024-03-25T23:36:44.617 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:36:44.617 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:36:44.617 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836532 2024-03-25T23:36:44.617 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:36:44.618 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836532 2024-03-25T23:36:44.618 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836532' 2024-03-25T23:36:44.618 INFO:tasks.workunit.client.0.smithi129.stderr:156881: waiting osd.0 seq 21474836532 2024-03-25T23:36:44.618 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:36:44.618 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836532 -lt 21474836532 2024-03-25T23:36:44.619 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:36:44.619 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-283467841539 2024-03-25T23:36:44.619 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:36:44.619 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:36:44.619 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-283467841539 2024-03-25T23:36:44.619 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:36:44.620 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841539 2024-03-25T23:36:44.620 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 283467841539' 2024-03-25T23:36:44.620 INFO:tasks.workunit.client.0.smithi129.stderr:156881: waiting osd.1 seq 283467841539 2024-03-25T23:36:44.620 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:36:45.256 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:221tell osd.2 flush_pg_stats 2024-03-25T23:36:45.256 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808834 2024-03-25T23:36:45.256 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808834 2024-03-25T23:36:45.256 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531 1-283467841538 2-287762808834' 2024-03-25T23:36:45.256 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:36:45.256 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:36:45.256 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070470 2024-03-25T23:36:45.256 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070470 2024-03-25T23:36:45.256 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531 1-283467841538 2-287762808834 3-253403070470' 2024-03-25T23:36:45.256 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:36:45.256 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836531 2024-03-25T23:36:45.257 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:36:45.257 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:36:45.257 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836531 2024-03-25T23:36:45.257 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:36:45.257 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836531 2024-03-25T23:36:45.257 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836531' 2024-03-25T23:36:45.257 INFO:tasks.workunit.client.0.smithi129.stderr:156878: waiting osd.0 seq 21474836531 2024-03-25T23:36:45.257 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:36:45.257 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836530 -lt 21474836531 2024-03-25T23:36:45.257 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:36:45.257 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:36:45.257 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:36:45.257 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836532 -lt 21474836531 2024-03-25T23:36:45.257 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:36:45.257 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-283467841538 2024-03-25T23:36:45.258 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:36:45.258 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:36:45.258 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-283467841538 2024-03-25T23:36:45.258 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:36:45.258 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841538 2024-03-25T23:36:45.258 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 283467841538' 2024-03-25T23:36:45.258 INFO:tasks.workunit.client.0.smithi129.stderr:156878: waiting osd.1 seq 283467841538 2024-03-25T23:36:45.258 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:36:45.258 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841539 -lt 283467841538 2024-03-25T23:36:45.258 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:36:46.719 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-287762808834 2024-03-25T23:36:46.720 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:36:46.720 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:36:46.720 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-287762808834 2024-03-25T23:36:46.720 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:36:46.720 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808834 2024-03-25T23:36:46.720 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 287762808834' 2024-03-25T23:36:46.720 INFO:tasks.workunit.client.0.smithi129.stderr:156878: waiting osd.2 seq 287762808834 2024-03-25T23:36:46.720 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:36:46.720 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808834 -lt 287762808834 2024-03-25T23:36:46.720 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:36:46.720 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070470 2024-03-25T23:36:46.721 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:36:46.721 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:36:46.721 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070470 2024-03-25T23:36:46.721 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:36:46.721 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070470 2024-03-25T23:36:46.721 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070470' 2024-03-25T23:36:46.721 INFO:tasks.workunit.client.0.smithi129.stderr:156878: waiting osd.3 seq 253403070470 2024-03-25T23:36:46.721 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:36:46.721 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070471 -lt 253403070470 2024-03-25T23:36:46.721 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:36:46.721 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:36:46.721 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:36:46.722 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:36:46.722 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:36:46.722 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:36:46.722 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:36:46.722 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:36:46.722 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:36:46.722 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:36:46.722 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:36:46.722 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-25T23:36:46.722 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:36:46.933 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.c4: flush_pg_stats: test 283467841539 -lt 283467841539 2024-03-25T23:36:46.933 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:36:46.933 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-287762808835 2024-03-25T23:36:46.933 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:36:46.933 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:36:46.933 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-287762808835 2024-03-25T23:36:46.933 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:36:46.933 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808835 2024-03-25T23:36:46.933 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 287762808835' 2024-03-25T23:36:46.934 INFO:tasks.workunit.client.0.smithi129.stderr:156881: waiting osd.2 seq 287762808835 2024-03-25T23:36:46.934 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:36:46.934 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808834 -lt 287762808835 2024-03-25T23:36:46.934 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:36:46.934 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:36:46.934 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:36:46.934 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808835 -lt 287762808835 2024-03-25T23:36:46.934 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:36:46.934 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070471 2024-03-25T23:36:46.934 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:36:46.934 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:36:46.934 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070471 2024-03-25T23:36:46.934 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:36:46.934 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070471 2024-03-25T23:36:46.935 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070471' 2024-03-25T23:36:46.935 INFO:tasks.workunit.client.0.smithi129.stderr:156881: waiting osd.3 seq 253403070471 2024-03-25T23:36:46.935 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:36:46.935 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070471 -lt 253403070471 2024-03-25T23:36:46.935 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:36:46.935 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:36:46.935 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:36:46.935 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:36:46.935 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:36:46.935 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:36:46.935 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:36:46.935 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:36:48.256 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(containslient.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:36:48.256 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:36:48.256 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-25T23:36:48.256 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-25T23:36:48.256 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-25T23:36:48.256 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-25T23:36:48.257 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:36:48.257 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-25T23:36:48.257 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:36:48.257 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:36:48.257 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:36:48.257 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:36:48.257 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:36:48.257 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:36:48.257 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:36:48.257 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:36:48.257 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-25T23:36:48.257 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:36:48.257 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:36:48.257 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:36:48.258 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-25T23:36:48.258 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-25T23:36:48.258 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-25T23:36:48.258 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-25T23:36:48.258 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-25T23:36:48.258 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-25T23:36:48.258 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-25T23:36:48.258 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-25T23:36:48.258 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-25T23:36:48.258 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-25T23:36:48.258 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-25T23:36:48.432 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/("stale") | not)' 2024-03-25T23:36:48.433 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:36:48.433 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:36:48.433 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-25T23:36:48.433 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:36:48.433 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:36:48.433 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:36:48.433 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-25T23:36:48.433 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-25T23:36:48.433 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-25T23:36:48.433 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-25T23:36:48.433 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:36:48.434 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-25T23:36:48.434 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:36:48.434 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:36:48.434 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:36:48.434 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:36:48.434 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:36:48.434 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:36:48.434 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:36:48.434 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:36:48.434 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:36:48.434 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:36:48.434 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:36:48.435 INFO:tasks.workunit.client.0.smithi129.stderr:156881: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:36:48.435 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:36:48.435 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:36:48.435 INFO:tasks.workunit.client.0.smithi129.stderr:156881: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:36:48.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:36:49.111 INFO:tasks.workunit.client.0.smithi129.stderr:clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-25T23:36:49.111 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:36:49.111 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-25T23:36:49.111 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:36:49.111 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:36:49.112 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:36:49.112 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:36:49.112 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:36:49.112 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:36:49.112 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:36:49.112 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:36:49.112 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:36:49.112 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:36:49.112 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:36:49.112 INFO:tasks.workunit.client.0.smithi129.stderr:156878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:36:49.112 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:36:49.112 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:36:49.112 INFO:tasks.workunit.client.0.smithi129.stderr:156878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:36:49.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:36:49.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-25T23:36:49.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 156879 2024-03-25T23:36:49.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-25T23:36:49.113 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-25T23:36:49.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-25T23:36:49.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-03-25T23:36:49.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-25T23:36:49.113 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-03-25T23:36:49.113 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-25T23:36:49.113 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-25T23:36:49.113 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-25T23:36:49.113 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-25T23:36:49.442 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-03-25T23:36:49.442 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-03-25T23:36:49.442 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-25T23:36:49.443 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-25T23:36:49.443 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:36:49.443 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:36:49.443 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:36:49.444 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:36:49.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-25T23:36:20.037815+0000 2024-03-25T23:36:49.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-25T23:36:50.034 INFO:tasks.workunit.client.0.smithi129.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-25T23:36:50.051 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-25T23:36:20.037815+0000 2024-03-25T23:36:50.051 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-25T23:36:50.051 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-25T23:36:20.037815+0000 2024-03-25T23:36:50.051 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-25T23:36:50.052 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-25T23:36:50.052 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:36:50.052 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:36:50.053 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:36:50.053 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:36:50.053 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:36:50.053 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:36:50.354 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:36:20.037815+0000 '>' 2024-03-25T23:36:20.037815+0000 2024-03-25T23:36:50.355 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:36:51.356 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:36:51.356 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:36:51.357 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:36:51.357 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:36:51.357 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:36:51.357 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:36:51.358 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:36:51.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:36:20.037815+0000 '>' 2024-03-25T23:36:20.037815+0000 2024-03-25T23:36:51.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:36:52.690 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:36:52.690 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:36:52.691 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:36:52.692 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:36:52.692 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:36:52.692 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:36:52.692 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:36:53.008 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:36:20.037815+0000 '>' 2024-03-25T23:36:20.037815+0000 2024-03-25T23:36:53.008 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:36:54.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:36:54.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:36:54.011 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:36:54.011 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:36:54.011 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:36:54.011 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:36:54.011 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:36:54.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:36:51.008751+0000 '>' 2024-03-25T23:36:20.037815+0000 2024-03-25T23:36:54.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-25T23:36:54.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-03-25T23:36:54.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-25T23:36:54.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:36:54.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:36:54.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 159898"' 2024-03-25T23:36:54.332 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 159898' 2024-03-25T23:36:54.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-03-25T23:36:54.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:36:54.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:36:54.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-25T23:36:54.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/159900: /' 2024-03-25T23:36:54.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 159901"' 2024-03-25T23:36:54.332 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 159901' 2024-03-25T23:36:54.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-03-25T23:36:54.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 159898 159901' 2024-03-25T23:36:54.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-25T23:36:54.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-25T23:36:54.333 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 159898 2024-03-25T23:36:54.333 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-03-25T23:36:54.333 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/159903: /' 2024-03-25T23:36:55.129 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:36:55.129 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:36:55.130 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-25T23:36:55.130 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:36:55.130 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-25T23:36:55.130 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:36:55.130 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:36:55.130 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-25T23:36:55.130 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:36:55.130 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-25T23:36:55.130 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:36:55.130 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:36:55.130 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:36:55.130 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:36:55.131 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:36:55.131 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:36:55.131 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-25T23:36:55.131 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:36:55.131 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:36:55.131 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-25T23:36:55.131 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:36:55.131 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:36:55.131 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-25T23:36:55.131 INFO:tasks.workunit.client.0.smithi129.stderr:159900: _ 2024-03-25T23:36:55.131 INFO:tasks.workunit.client.0.smithi129.stderr:159900: hinfo_key 2024-03-25T23:36:55.131 INFO:tasks.workunit.client.0.smithi129.stderr:159900: snapset 2024-03-25T23:36:55.131 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-25T23:36:55.131 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:36:55.132 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:36:55.132 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-25T23:36:55.132 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:36:55.132 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:36:55.132 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:36:55.132 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:36:55.134 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:36:55.134 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:36:55.134 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-25T23:36:55.134 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-25T23:36:55.134 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:36:55.134 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:36:55.134 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:36:55.134 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:36:55.134 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:36:55.134 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:36:55.134 INFO:tasks.workunit.client.0.smithi129.stderr:159900: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:36:55.134 INFO:tasks.workunit.client.0.smithi129.stderr:159900: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:36:55.134 INFO:tasks.workunit.client.0.smithi129.stderr:159900: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:36:55.135 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:36:55.135 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:36:55.135 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:36:55.135 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:36:55.135 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:36:55.135 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:36:55.135 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:36:55.135 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:36:55.135 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:36:55.135 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:36:55.135 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:36:55.135 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-25T23:36:55.135 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-25T23:36:55.135 INFO:tasks.workunit.client.0.smithi129.stderr:159900: start osd.1 2024-03-25T23:36:55.191 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:36:55.191 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:36:55.191 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-03-25T23:36:55.191 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:36:55.191 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2024-03-25T23:36:55.191 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:36:55.191 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:36:55.191 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-03-25T23:36:55.192 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:36:55.192 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-25T23:36:55.192 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:36:55.192 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:36:55.192 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:36:55.192 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:36:55.192 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:36:55.192 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:36:55.192 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2024-03-25T23:36:55.192 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:36:55.192 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:36:55.192 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-03-25T23:36:55.192 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:36:55.192 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-03-25T23:36:55.193 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2024-03-25T23:36:55.193 INFO:tasks.workunit.client.0.smithi129.stderr:159903: _ 2024-03-25T23:36:55.193 INFO:tasks.workunit.client.0.smithi129.stderr:159903: hinfo_key 2024-03-25T23:36:55.193 INFO:tasks.workunit.client.0.smithi129.stderr:159903: snapset 2024-03-25T23:36:55.193 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-03-25T23:36:55.193 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:36:55.193 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:36:55.193 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-25T23:36:55.193 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:36:55.193 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-25T23:36:55.193 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:36:55.193 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:36:55.194 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:36:55.194 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:36:55.194 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-25T23:36:55.194 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-25T23:36:55.194 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:36:55.194 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:36:55.194 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:36:55.194 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:36:55.194 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:36:55.195 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:36:55.195 INFO:tasks.workunit.client.0.smithi129.stderr:159903: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:36:55.195 INFO:tasks.workunit.client.0.smithi129.stderr:159903: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:36:55.195 INFO:tasks.workunit.client.0.smithi129.stderr:159903: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:36:55.195 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:36:55.195 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:36:55.195 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:36:55.195 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:36:55.195 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:36:55.195 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:36:55.195 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:36:55.195 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:36:55.195 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:36:55.195 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:36:55.196 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:36:55.196 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-25T23:36:55.196 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-25T23:36:55.196 INFO:tasks.workunit.client.0.smithi129.stderr:159903: start osd.2 2024-03-25T23:36:59.641 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:36:59.642 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-25T23:36:59.642 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-25T23:36:59.642 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:36:59.642 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:36:59.642 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:36:59.642 INFO:tasks.workunit.client.0.smithi129.stderr:159903: 2024-03-25T23:36:55.218+0000 7f34249037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:36:59.642 INFO:tasks.workunit.client.0.smithi129.stderr:159903: 2024-03-25T23:36:55.250+0000 7f34249037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:36:59.642 INFO:tasks.workunit.client.0.smithi129.stderr:159903: 2024-03-25T23:36:55.250+0000 7f34249037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:36:59.642 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-25T23:36:59.642 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:36:59.642 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-25T23:36:59.642 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:36:59.643 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:36:59.643 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:36:59.643 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:36:59.643 INFO:tasks.workunit.client.0.smithi129.stderr:159903: 0 2024-03-25T23:36:59.643 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:36:59.643 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:36:59.643 INFO:tasks.workunit.client.0.smithi129.stderr:159903: 2024-03-25T23:36:55.802+0000 7f34249037c0 -1 Falling back to public interface 2024-03-25T23:36:59.643 INFO:tasks.workunit.client.0.smithi129.stderr:159903: 2024-03-25T23:36:55.802+0000 7f34238c3640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6810/0: (98) Address already in use 2024-03-25T23:36:59.643 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:36:59.643 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:36:59.643 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:36:59.643 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:36:59.643 INFO:tasks.workunit.client.0.smithi129.stderr:159903: 1 2024-03-25T23:36:59.643 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:36:59.644 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:36:59.644 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:36:59.644 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:36:59.644 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:36:59.644 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:36:59.644 INFO:tasks.workunit.client.0.smithi129.stderr:159903: 2 2024-03-25T23:36:59.644 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:36:59.644 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:36:59.644 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:36:59.644 INFO:tasks.workunit.client.0.smithi129.stderr:159903: 2024-03-25T23:36:59.638+0000 7f34249037c0 -1 osd.2 70 log_to_monitors true 2024-03-25T23:36:59.644 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:36:59.644 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:36:59.644 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:36:59.644 INFO:tasks.workunit.client.0.smithi129.stderr:159903: 3 2024-03-25T23:36:59.644 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:36:59.768 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/cloovery_ops 2024-03-25T23:36:59.768 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-25T23:36:59.768 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-25T23:36:59.768 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:36:59.768 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:36:59.768 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:36:59.768 INFO:tasks.workunit.client.0.smithi129.stderr:159900: 2024-03-25T23:36:55.166+0000 7f6f514bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:36:59.768 INFO:tasks.workunit.client.0.smithi129.stderr:159900: 2024-03-25T23:36:55.250+0000 7f6f514bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:36:59.769 INFO:tasks.workunit.client.0.smithi129.stderr:159900: 2024-03-25T23:36:55.250+0000 7f6f514bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:36:59.769 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-25T23:36:59.769 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:36:59.769 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T23:36:59.769 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:36:59.769 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:36:59.769 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:36:59.769 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:36:59.769 INFO:tasks.workunit.client.0.smithi129.stderr:159900: 0 2024-03-25T23:36:59.769 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:36:59.769 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:36:59.769 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:36:59.769 INFO:tasks.workunit.client.0.smithi129.stderr:159900: 2024-03-25T23:36:55.802+0000 7f6f514bf7c0 -1 Falling back to public interface 2024-03-25T23:36:59.769 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:36:59.770 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:36:59.770 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:36:59.770 INFO:tasks.workunit.client.0.smithi129.stderr:159900: 1 2024-03-25T23:36:59.770 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:36:59.770 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:36:59.770 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:36:59.770 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:36:59.770 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:36:59.770 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:36:59.770 INFO:tasks.workunit.client.0.smithi129.stderr:159900: 2 2024-03-25T23:36:59.770 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:36:59.770 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:36:59.770 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:36:59.770 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:36:59.771 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:36:59.771 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:36:59.771 INFO:tasks.workunit.client.0.smithi129.stderr:159900: 3 2024-03-25T23:36:59.771 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:36:59.771 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:36:59.771 INFO:tasks.workunit.client.0.smithi129.stderr:159900: 2024-03-25T23:36:59.598+0000 7f6f514bf7c0 -1 osd.1 70 log_to_monitors true 2024-03-25T23:37:02.123 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/cne.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:37:02.124 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:37:02.124 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:37:02.124 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:37:02.124 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:37:02.125 INFO:tasks.workunit.client.0.smithi129.stderr:159903: 4 2024-03-25T23:37:02.125 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:37:02.125 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:37:02.125 INFO:tasks.workunit.client.0.smithi129.stderr:159903: osd.2 up in weight 1 up_from 74 up_thru 49 down_at 71 last_clean_interval [67,70) [v2:127.0.0.1:6811/3735543029,v1:127.0.0.1:6813/3735543029] [v2:127.0.0.1:6815/3735543029,v1:127.0.0.1:6817/3735543029] exists,up c816e86b-623d-482b-83dd-84132fd7b4ae 2024-03-25T23:37:02.125 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:37:02.125 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:37:02.125 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:37:02.125 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:37:02.125 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:37:02.125 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:37:02.125 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:37:02.126 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:37:02.126 INFO:tasks.workunit.client.0.smithi129.stderr:159903: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:37:02.126 INFO:tasks.workunit.client.0.smithi129.stderr:159903: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:37:02.126 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:37:02.126 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:37:02.126 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:37:02.126 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:37:02.126 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:37:02.126 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:37:02.126 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:37:02.126 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:37:02.127 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:37:02.127 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:37:02.127 INFO:tasks.workunit.client.0.smithi129.stderr:159903: 1 2024-03-25T23:37:02.127 INFO:tasks.workunit.client.0.smithi129.stderr:159903: 2 2024-03-25T23:37:02.127 INFO:tasks.workunit.client.0.smithi129.stderr:159903: 3' 2024-03-25T23:37:02.127 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:37:02.127 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:37:02.127 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:37:02.127 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836537 2024-03-25T23:37:02.127 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836537 2024-03-25T23:37:02.127 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836537' 2024-03-25T23:37:02.128 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/cloneph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:37:02.128 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:37:02.128 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:37:02.129 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:37:02.129 INFO:tasks.workunit.client.0.smithi129.stderr:159900: 4 2024-03-25T23:37:02.129 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:37:02.129 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:37:02.129 INFO:tasks.workunit.client.0.smithi129.stderr:159900: osd.1 up in weight 1 up_from 74 up_thru 74 down_at 71 last_clean_interval [66,70) [v2:127.0.0.1:6810/4154047206,v1:127.0.0.1:6812/4154047206] [v2:127.0.0.1:6814/4154047206,v1:127.0.0.1:6816/4154047206] exists,up 0a1d267b-774d-4fda-8cc0-7d86cd87d221 2024-03-25T23:37:02.129 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:37:02.129 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:37:02.129 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:37:02.129 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:37:02.129 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:37:02.129 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:37:02.130 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:37:02.130 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:37:02.130 INFO:tasks.workunit.client.0.smithi129.stderr:159900: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:37:02.130 INFO:tasks.workunit.client.0.smithi129.stderr:159900: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:37:02.130 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:37:02.130 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:37:02.130 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:37:02.130 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:37:02.130 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:37:02.130 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:37:02.130 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:37:02.130 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:37:02.131 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:37:02.131 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:37:02.131 INFO:tasks.workunit.client.0.smithi129.stderr:159900: 1 2024-03-25T23:37:02.131 INFO:tasks.workunit.client.0.smithi129.stderr:159900: 2 2024-03-25T23:37:02.131 INFO:tasks.workunit.client.0.smithi129.stderr:159900: 3' 2024-03-25T23:37:02.131 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:37:02.131 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:37:02.131 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:37:02.131 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836538 2024-03-25T23:37:02.131 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836538 2024-03-25T23:37:02.131 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836538' 2024-03-25T23:37:02.131 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:37:04.196 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:37:04.196 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:37:04.196 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579906 2024-03-25T23:37:04.196 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579906 2024-03-25T23:37:04.196 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836537 1-317827579906' 2024-03-25T23:37:04.196 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:37:04.196 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:37:04.196 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579906 2024-03-25T23:37:04.196 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579906 2024-03-25T23:37:04.196 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836537 1-317827579906 2-317827579906' 2024-03-25T23:37:04.196 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:37:04.197 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:37:04.197 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070477 2024-03-25T23:37:04.197 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070477 2024-03-25T23:37:04.197 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836537 1-317827579906 2-317827579906 3-253403070477' 2024-03-25T23:37:04.197 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:37:04.197 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836537 2024-03-25T23:37:04.197 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:37:04.197 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:37:04.197 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836537 2024-03-25T23:37:04.197 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:37:04.197 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836537 2024-03-25T23:37:04.197 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836537' 2024-03-25T23:37:04.197 INFO:tasks.workunit.client.0.smithi129.stderr:159903: waiting osd.0 seq 21474836537 2024-03-25T23:37:04.197 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:37:04.198 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836536 -lt 21474836537 2024-03-25T23:37:04.198 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:37:04.198 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:37:04.198 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:37:04.198 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836538 -lt 21474836537 2024-03-25T23:37:04.198 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:37:04.198 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-317827579906 2024-03-25T23:37:04.198 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:37:04.198 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:37:04.205 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: e/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:37:04.205 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579907 2024-03-25T23:37:04.205 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579907 2024-03-25T23:37:04.205 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836538 1-317827579907' 2024-03-25T23:37:04.206 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:37:04.206 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:37:04.206 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579907 2024-03-25T23:37:04.206 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579907 2024-03-25T23:37:04.206 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836538 1-317827579907 2-317827579907' 2024-03-25T23:37:04.206 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:37:04.206 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:37:04.206 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070476 2024-03-25T23:37:04.206 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070476 2024-03-25T23:37:04.206 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836538 1-317827579907 2-317827579907 3-253403070476' 2024-03-25T23:37:04.206 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:37:04.206 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836538 2024-03-25T23:37:04.206 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:37:04.207 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:37:04.207 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836538 2024-03-25T23:37:04.207 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:37:04.207 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836538 2024-03-25T23:37:04.207 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836538' 2024-03-25T23:37:04.207 INFO:tasks.workunit.client.0.smithi129.stderr:159900: waiting osd.0 seq 21474836538 2024-03-25T23:37:04.207 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:37:04.207 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836536 -lt 21474836538 2024-03-25T23:37:04.207 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:37:04.207 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:37:04.207 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:37:04.207 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836538 -lt 21474836538 2024-03-25T23:37:04.208 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:37:04.208 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-317827579907 2024-03-25T23:37:04.208 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:37:04.208 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:37:04.208 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-317827579907 2024-03-25T23:37:04.208 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:37:05.539 INFO:tasks.workunit.client.0.smithi129.stderr:15990echo 1-317827579906 2024-03-25T23:37:05.539 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:37:05.539 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579906 2024-03-25T23:37:05.539 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 317827579906' 2024-03-25T23:37:05.539 INFO:tasks.workunit.client.0.smithi129.stderr:159903: waiting osd.1 seq 317827579906 2024-03-25T23:37:05.540 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:37:05.540 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579907 -lt 317827579906 2024-03-25T23:37:05.540 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:37:05.540 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-317827579906 2024-03-25T23:37:05.540 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:37:05.540 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:37:05.540 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-317827579906 2024-03-25T23:37:05.540 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:37:05.540 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579906 2024-03-25T23:37:05.540 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 317827579906' 2024-03-25T23:37:05.540 INFO:tasks.workunit.client.0.smithi129.stderr:159903: waiting osd.2 seq 317827579906 2024-03-25T23:37:05.540 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:37:05.540 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579907 -lt 317827579906 2024-03-25T23:37:05.541 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:37:05.541 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070477 2024-03-25T23:37:05.541 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:37:05.541 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:37:05.541 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070477 2024-03-25T23:37:05.541 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:37:05.541 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070477 2024-03-25T23:37:05.541 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070477' 2024-03-25T23:37:05.541 INFO:tasks.workunit.client.0.smithi129.stderr:159903: waiting osd.3 seq 253403070477 2024-03-25T23:37:05.541 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:37:05.541 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070477 -lt 253403070477 2024-03-25T23:37:05.541 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:37:05.541 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:37:05.541 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:37:05.541 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:37:05.541 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:37:05.542 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:37:05.542 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:37:05.589 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("0: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579907 2024-03-25T23:37:05.589 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 317827579907' 2024-03-25T23:37:05.589 INFO:tasks.workunit.client.0.smithi129.stderr:159900: waiting osd.1 seq 317827579907 2024-03-25T23:37:05.589 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:37:05.590 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579907 -lt 317827579907 2024-03-25T23:37:05.590 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:37:05.590 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-317827579907 2024-03-25T23:37:05.590 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:37:05.590 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:37:05.590 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-317827579907 2024-03-25T23:37:05.590 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:37:05.590 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579907 2024-03-25T23:37:05.590 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 317827579907' 2024-03-25T23:37:05.590 INFO:tasks.workunit.client.0.smithi129.stderr:159900: waiting osd.2 seq 317827579907 2024-03-25T23:37:05.590 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:37:05.590 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579907 -lt 317827579907 2024-03-25T23:37:05.591 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:37:05.591 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070476 2024-03-25T23:37:05.591 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:37:05.591 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:37:05.591 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070476 2024-03-25T23:37:05.591 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:37:05.591 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070476 2024-03-25T23:37:05.591 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070476' 2024-03-25T23:37:05.591 INFO:tasks.workunit.client.0.smithi129.stderr:159900: waiting osd.3 seq 253403070476 2024-03-25T23:37:05.591 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:37:05.591 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070477 -lt 253403070476 2024-03-25T23:37:05.591 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:37:05.591 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:37:05.591 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:37:05.591 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:37:05.592 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:37:05.592 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:37:05.592 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:37:05.592 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:37:06.220 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:37:06.220 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:37:06.221 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:37:06.221 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:37:06.221 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:37:06.221 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:37:06.221 INFO:tasks.workunit.client.0.smithi129.stderr:159900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:37:06.221 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:37:06.221 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:37:06.221 INFO:tasks.workunit.client.0.smithi129.stderr:159900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:37:06.221 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:37:06.221 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-25T23:37:06.221 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 159901 2024-03-25T23:37:06.228 INFO:tasks.workunit.client.0.smithi129.stderr:clean")) | ' 2024-03-25T23:37:06.228 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:37:06.228 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:37:06.228 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:37:06.228 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:37:06.228 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:37:06.228 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:37:06.228 INFO:tasks.workunit.client.0.smithi129.stderr:159903: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:37:06.228 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:37:06.228 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:37:06.228 INFO:tasks.workunit.client.0.smithi129.stderr:159903: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:37:06.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:37:06.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-25T23:37:06.229 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-25T23:37:06.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-25T23:37:06.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-03-25T23:37:06.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-25T23:37:06.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-25T23:37:06.301 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-25T23:37:06.304 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2024-03-25T23:37:06.304 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-03-25T23:37:06.304 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-03-25T23:37:06.304 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-03-25T23:37:06.305 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=1 2024-03-25T23:37:06.305 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-03-25T23:37:06.305 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-25T23:37:06.305 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:37:06.305 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:37:06.305 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 162261"' 2024-03-25T23:37:06.305 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 162261' 2024-03-25T23:37:06.305 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-25T23:37:06.305 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:37:06.305 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:37:06.305 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 162263"' 2024-03-25T23:37:06.305 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-25T23:37:06.305 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 162263' 2024-03-25T23:37:06.306 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-03-25T23:37:06.306 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 162261 162263' 2024-03-25T23:37:06.306 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-25T23:37:06.306 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-25T23:37:06.306 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 162261 2024-03-25T23:37:06.306 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/162264: /' 2024-03-25T23:37:06.306 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-25T23:37:06.306 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/162266: /' 2024-03-25T23:37:09.551 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:37:09.551 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:37:09.551 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-25T23:37:09.551 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:37:09.551 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-03-25T23:37:09.551 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:37:09.552 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:37:09.552 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-25T23:37:09.552 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:37:09.552 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-25T23:37:09.552 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:37:09.552 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:37:09.552 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:37:09.552 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:37:09.552 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:37:09.552 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:37:09.552 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-03-25T23:37:09.552 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:37:09.552 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:37:09.552 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-25T23:37:09.552 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:37:09.553 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-25T23:37:09.553 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-03-25T23:37:09.553 INFO:tasks.workunit.client.0.smithi129.stderr:162264: remove 0#2:eb822e21:::SOMETHING:head# 2024-03-25T23:37:09.553 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-25T23:37:09.553 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:37:09.553 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:37:09.553 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-25T23:37:09.553 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:37:09.553 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-25T23:37:09.553 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:37:09.553 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:37:09.556 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:37:09.556 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:37:09.556 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-25T23:37:09.556 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-25T23:37:09.556 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:37:09.556 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:37:09.557 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:37:09.557 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:37:09.557 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:37:09.557 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:37:09.557 INFO:tasks.workunit.client.0.smithi129.stderr:162264: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:37:09.557 INFO:tasks.workunit.client.0.smithi129.stderr:162264: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:37:09.557 INFO:tasks.workunit.client.0.smithi129.stderr:162264: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:37:09.557 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:37:09.557 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:37:09.557 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:37:09.557 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:37:09.557 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:37:09.557 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:37:09.557 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:37:09.558 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:37:09.558 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:37:09.558 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:37:09.558 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:37:09.558 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-25T23:37:09.558 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-25T23:37:09.558 INFO:tasks.workunit.client.0.smithi129.stderr:162264: start osd.3 2024-03-25T23:37:09.558 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:37:09.948 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:37:09.948 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:37:09.948 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-25T23:37:09.949 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:37:09.949 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-25T23:37:09.949 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:37:09.949 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:37:09.949 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-25T23:37:09.949 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:37:09.949 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-25T23:37:09.949 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:37:09.949 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:37:09.949 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:37:09.949 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:37:09.949 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:37:09.949 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:37:09.950 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-25T23:37:09.950 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:37:09.950 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:37:09.950 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-25T23:37:09.950 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:37:09.950 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:37:09.950 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-25T23:37:09.950 INFO:tasks.workunit.client.0.smithi129.stderr:162266: remove 1#2:eb822e21:::SOMETHING:head# 2024-03-25T23:37:09.950 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-25T23:37:09.950 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:37:09.950 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:37:09.950 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-25T23:37:09.951 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:37:09.951 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:37:09.951 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:37:09.951 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:37:09.952 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:37:09.952 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:37:09.952 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-25T23:37:09.952 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-25T23:37:09.952 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:37:09.952 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:37:09.952 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:37:09.952 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:37:09.952 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:37:09.952 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:37:09.952 INFO:tasks.workunit.client.0.smithi129.stderr:162266: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:37:09.952 INFO:tasks.workunit.client.0.smithi129.stderr:162266: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:37:09.952 INFO:tasks.workunit.client.0.smithi129.stderr:162266: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:37:09.952 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:37:09.953 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:37:09.953 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:37:09.953 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:37:09.953 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:37:09.953 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:37:09.953 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:37:09.953 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:37:09.953 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:37:09.953 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:37:09.953 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:37:09.953 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-25T23:37:09.953 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-25T23:37:09.953 INFO:tasks.workunit.client.0.smithi129.stderr:162266: start osd.1 2024-03-25T23:37:09.953 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-25T23:37:14.404 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-25T23:37:14.404 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-25T23:37:14.404 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:37:14.404 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:37:14.404 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:37:14.404 INFO:tasks.workunit.client.0.smithi129.stderr:162264: 2024-03-25T23:37:09.586+0000 7f6f38b8c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:37:14.404 INFO:tasks.workunit.client.0.smithi129.stderr:162264: 2024-03-25T23:37:09.590+0000 7f6f38b8c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:37:14.405 INFO:tasks.workunit.client.0.smithi129.stderr:162264: 2024-03-25T23:37:09.590+0000 7f6f38b8c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:37:14.405 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-25T23:37:14.405 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:37:14.405 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-25T23:37:14.405 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:37:14.405 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:37:14.405 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:37:14.405 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:37:14.405 INFO:tasks.workunit.client.0.smithi129.stderr:162264: 0 2024-03-25T23:37:14.405 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:37:14.405 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:37:14.405 INFO:tasks.workunit.client.0.smithi129.stderr:162264: 2024-03-25T23:37:10.138+0000 7f6f38b8c7c0 -1 Falling back to public interface 2024-03-25T23:37:14.406 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:37:14.406 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:37:14.406 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:37:14.406 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:37:14.406 INFO:tasks.workunit.client.0.smithi129.stderr:162264: 1 2024-03-25T23:37:14.406 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:37:14.406 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:37:14.406 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:37:14.406 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:37:14.406 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:37:14.406 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:37:14.406 INFO:tasks.workunit.client.0.smithi129.stderr:162264: 2 2024-03-25T23:37:14.407 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:37:14.407 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:37:14.407 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:37:14.407 INFO:tasks.workunit.client.0.smithi129.stderr:162264: 2024-03-25T23:37:13.070+0000 7f6f38b8c7c0 -1 osd.3 77 log_to_monitors true 2024-03-25T23:37:14.407 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:37:14.407 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:37:14.407 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:37:14.407 INFO:tasks.workunit.client.0.smithi129.stderr:162264: 3 2024-03-25T23:37:14.407 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:37:14.407 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:37:14.765 INFO:tasks.workunit.client.0.smithi129.stderr:162264: osd.3 up in weight 1 up_from 82 up_thru 75 down_at 78 last_-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:37:14.765 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-25T23:37:14.765 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:37:14.765 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:37:14.765 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:37:14.765 INFO:tasks.workunit.client.0.smithi129.stderr:162266: 2024-03-25T23:37:09.982+0000 7fe8135387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:37:14.765 INFO:tasks.workunit.client.0.smithi129.stderr:162266: 2024-03-25T23:37:10.006+0000 7fe8135387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:37:14.765 INFO:tasks.workunit.client.0.smithi129.stderr:162266: 2024-03-25T23:37:10.006+0000 7fe8135387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:37:14.766 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-25T23:37:14.766 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:37:14.766 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T23:37:14.766 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:37:14.766 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:37:14.766 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:37:14.766 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:37:14.766 INFO:tasks.workunit.client.0.smithi129.stderr:162266: 0 2024-03-25T23:37:14.766 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:37:14.766 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:37:14.766 INFO:tasks.workunit.client.0.smithi129.stderr:162266: 2024-03-25T23:37:10.554+0000 7fe8135387c0 -1 Falling back to public interface 2024-03-25T23:37:14.766 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:37:14.766 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:37:14.767 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:37:14.767 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:37:14.767 INFO:tasks.workunit.client.0.smithi129.stderr:162266: 1 2024-03-25T23:37:14.767 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:37:14.767 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:37:14.767 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:37:14.767 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:37:14.767 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:37:14.767 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:37:14.767 INFO:tasks.workunit.client.0.smithi129.stderr:162266: 2 2024-03-25T23:37:14.767 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:37:14.768 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:37:14.768 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:37:14.768 INFO:tasks.workunit.client.0.smithi129.stderr:162266: 2024-03-25T23:37:13.838+0000 7fe8135387c0 -1 osd.1 77 log_to_monitors true 2024-03-25T23:37:14.768 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:37:14.768 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:37:14.768 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:37:14.768 INFO:tasks.workunit.client.0.smithi129.stderr:162266: 3 2024-03-25T23:37:14.768 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:37:14.768 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:37:15.959 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperclean_interval [59,77) [v2:127.0.0.1:6810/893959796,v1:127.0.0.1:6812/893959796] [v2:127.0.0.1:6814/893959796,v1:127.0.0.1:6816/893959796] exists,up 614c4a39-b13c-4192-ba13-0218e9f2bcd7 2024-03-25T23:37:15.959 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:37:15.959 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:37:15.959 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:37:15.959 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:37:15.960 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:37:15.960 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:37:15.960 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:37:15.960 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:37:15.960 INFO:tasks.workunit.client.0.smithi129.stderr:162264: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:37:15.960 INFO:tasks.workunit.client.0.smithi129.stderr:162264: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:37:15.960 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:37:15.960 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:37:15.960 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:37:15.960 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:37:15.960 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:37:15.960 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:37:15.960 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:37:15.960 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:37:15.960 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:37:15.961 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:37:15.961 INFO:tasks.workunit.client.0.smithi129.stderr:162264: 1 2024-03-25T23:37:15.961 INFO:tasks.workunit.client.0.smithi129.stderr:162264: 2 2024-03-25T23:37:15.961 INFO:tasks.workunit.client.0.smithi129.stderr:162264: 3' 2024-03-25T23:37:15.961 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:37:15.961 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:37:15.961 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:37:15.961 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836541 2024-03-25T23:37:15.961 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836541 2024-03-25T23:37:15.961 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541' 2024-03-25T23:37:15.961 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:37:15.961 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:37:15.961 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=2 2024-03-25T23:37:15.961 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 2 2024-03-25T23:37:15.962 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541 1-2' 2024-03-25T23:37:15.962 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:37:15.962 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:37:17.043 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /hos.sh:985: wait_for_osd: sleep 1 2024-03-25T23:37:17.043 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:37:17.043 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:37:17.043 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:37:17.043 INFO:tasks.workunit.client.0.smithi129.stderr:162266: 4 2024-03-25T23:37:17.043 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:37:17.043 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:37:17.043 INFO:tasks.workunit.client.0.smithi129.stderr:162266: osd.1 up in weight 1 up_from 85 up_thru 85 down_at 78 last_clean_interval [74,77) [v2:127.0.0.1:6826/4073472964,v1:127.0.0.1:6827/4073472964] [v2:127.0.0.1:6828/4073472964,v1:127.0.0.1:6829/4073472964] exists,up 0a1d267b-774d-4fda-8cc0-7d86cd87d221 2024-03-25T23:37:17.043 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:37:17.043 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:37:17.044 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:37:17.044 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:37:17.044 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:37:17.044 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:37:17.044 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:37:17.044 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:37:17.044 INFO:tasks.workunit.client.0.smithi129.stderr:162266: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:37:17.044 INFO:tasks.workunit.client.0.smithi129.stderr:162266: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:37:17.044 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:37:17.044 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:37:17.044 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:37:17.044 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:37:17.044 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:37:17.044 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:37:17.045 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:37:17.045 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:37:17.045 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:37:17.045 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:37:17.045 INFO:tasks.workunit.client.0.smithi129.stderr:162266: 1 2024-03-25T23:37:17.045 INFO:tasks.workunit.client.0.smithi129.stderr:162266: 2 2024-03-25T23:37:17.045 INFO:tasks.workunit.client.0.smithi129.stderr:162266: 3' 2024-03-25T23:37:17.045 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:37:17.045 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:37:17.045 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:37:17.045 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836543 2024-03-25T23:37:17.045 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836543 2024-03-25T23:37:17.045 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543' 2024-03-25T23:37:17.045 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:37:17.326 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helme/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579911 2024-03-25T23:37:17.326 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579911 2024-03-25T23:37:17.326 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541 1-2 2-317827579911' 2024-03-25T23:37:17.326 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:37:17.326 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:37:17.326 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318274 2024-03-25T23:37:17.326 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318274 2024-03-25T23:37:17.327 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541 1-2 2-317827579911 3-352187318274' 2024-03-25T23:37:17.327 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:37:17.327 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836541 2024-03-25T23:37:17.327 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:37:17.327 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:37:17.327 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836541 2024-03-25T23:37:17.327 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:37:17.327 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836541 2024-03-25T23:37:17.327 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836541' 2024-03-25T23:37:17.327 INFO:tasks.workunit.client.0.smithi129.stderr:162264: waiting osd.0 seq 21474836541 2024-03-25T23:37:17.327 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:37:17.327 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836541 -lt 21474836541 2024-03-25T23:37:17.327 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:37:17.327 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:37:17.328 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-2 2024-03-25T23:37:17.328 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:37:17.328 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-2 2024-03-25T23:37:17.328 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:37:17.328 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=2 2024-03-25T23:37:17.328 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 2' 2024-03-25T23:37:17.328 INFO:tasks.workunit.client.0.smithi129.stderr:162264: waiting osd.1 seq 2 2024-03-25T23:37:17.328 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:37:17.328 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 2 -lt 2 2024-03-25T23:37:17.328 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:37:17.328 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:37:17.328 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-317827579911 2024-03-25T23:37:17.328 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:37:17.328 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-317827579911 2024-03-25T23:37:17.329 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:37:19.138 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/spers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:37:19.138 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=365072220163 2024-03-25T23:37:19.138 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 365072220163 2024-03-25T23:37:19.138 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543 1-365072220163' 2024-03-25T23:37:19.138 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:37:19.138 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:37:19.138 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579912 2024-03-25T23:37:19.139 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579912 2024-03-25T23:37:19.139 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543 1-365072220163 2-317827579912' 2024-03-25T23:37:19.139 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:37:19.139 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:37:19.139 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318275 2024-03-25T23:37:19.139 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318275 2024-03-25T23:37:19.139 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543 1-365072220163 2-317827579912 3-352187318275' 2024-03-25T23:37:19.139 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:37:19.139 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836543 2024-03-25T23:37:19.139 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:37:19.139 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:37:19.139 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836543 2024-03-25T23:37:19.139 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:37:19.139 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836543 2024-03-25T23:37:19.140 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836543' 2024-03-25T23:37:19.140 INFO:tasks.workunit.client.0.smithi129.stderr:162266: waiting osd.0 seq 21474836543 2024-03-25T23:37:19.140 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:37:19.140 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836541 -lt 21474836543 2024-03-25T23:37:19.140 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:37:19.140 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:37:19.140 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:37:19.140 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836543 -lt 21474836543 2024-03-25T23:37:19.140 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:37:19.140 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-365072220163 2024-03-25T23:37:19.140 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:37:19.140 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:37:19.140 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-365072220163 2024-03-25T23:37:19.140 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:37:19.956 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/utandalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579911 2024-03-25T23:37:19.956 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 317827579911' 2024-03-25T23:37:19.956 INFO:tasks.workunit.client.0.smithi129.stderr:162264: waiting osd.2 seq 317827579911 2024-03-25T23:37:19.956 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:37:19.956 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579910 -lt 317827579911 2024-03-25T23:37:19.956 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:37:19.956 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:37:19.956 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:37:19.956 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579912 -lt 317827579911 2024-03-25T23:37:19.956 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:37:19.957 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-352187318274 2024-03-25T23:37:19.957 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:37:19.957 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:37:19.957 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-352187318274 2024-03-25T23:37:19.957 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:37:19.957 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318274 2024-03-25T23:37:19.957 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 352187318274' 2024-03-25T23:37:19.957 INFO:tasks.workunit.client.0.smithi129.stderr:162264: waiting osd.3 seq 352187318274 2024-03-25T23:37:19.957 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:37:19.957 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318275 -lt 352187318274 2024-03-25T23:37:19.957 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:37:19.957 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:37:19.957 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:37:19.957 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:37:19.958 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:37:19.958 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:37:19.958 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:37:19.958 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:37:19.958 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:37:19.958 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:37:19.958 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:37:19.958 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T23:37:19.958 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:37:19.958 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:37:20.572 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/sbuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=365072220163 2024-03-25T23:37:20.572 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 365072220163' 2024-03-25T23:37:20.573 INFO:tasks.workunit.client.0.smithi129.stderr:162266: waiting osd.1 seq 365072220163 2024-03-25T23:37:20.573 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:37:20.573 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 365072220163 -lt 365072220163 2024-03-25T23:37:20.573 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:37:20.573 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-317827579912 2024-03-25T23:37:20.573 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:37:20.573 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:37:20.573 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-317827579912 2024-03-25T23:37:20.573 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:37:20.573 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579912 2024-03-25T23:37:20.573 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 317827579912' 2024-03-25T23:37:20.573 INFO:tasks.workunit.client.0.smithi129.stderr:162266: waiting osd.2 seq 317827579912 2024-03-25T23:37:20.573 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:37:20.573 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579912 -lt 317827579912 2024-03-25T23:37:20.574 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:37:20.574 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-352187318275 2024-03-25T23:37:20.574 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:37:20.574 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:37:20.574 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-352187318275 2024-03-25T23:37:20.574 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:37:20.574 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318275 2024-03-25T23:37:20.574 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 352187318275' 2024-03-25T23:37:20.574 INFO:tasks.workunit.client.0.smithi129.stderr:162266: waiting osd.3 seq 352187318275 2024-03-25T23:37:20.574 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:37:20.574 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318276 -lt 352187318275 2024-03-25T23:37:20.574 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:37:20.574 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:37:20.574 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:37:20.574 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:37:20.575 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:37:20.575 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:37:20.575 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:37:20.575 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:37:21.485 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("tandalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:37:21.485 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-25T23:37:21.486 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-25T23:37:21.486 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-25T23:37:21.486 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-25T23:37:21.486 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:37:21.486 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-25T23:37:21.486 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:37:21.486 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:37:21.486 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:37:21.486 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:37:21.486 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:37:21.486 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:37:21.486 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:37:21.487 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:37:21.487 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T23:37:21.487 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:37:21.487 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:37:21.487 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:37:21.487 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-25T23:37:21.487 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-25T23:37:21.487 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-25T23:37:21.487 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-25T23:37:21.487 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-25T23:37:21.487 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-25T23:37:21.487 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-25T23:37:21.487 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-25T23:37:21.487 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-25T23:37:21.488 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1 2024-03-25T23:37:21.488 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1 '!=' null 2024-03-25T23:37:21.488 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-03-25T23:37:22.012 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-hstale") | not)' 2024-03-25T23:37:22.013 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:37:22.013 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:37:22.013 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T23:37:22.013 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:37:22.013 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:37:22.013 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:37:22.013 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-25T23:37:22.013 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-25T23:37:22.013 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-25T23:37:22.013 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-25T23:37:22.013 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:37:22.013 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-25T23:37:22.014 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:37:22.014 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:37:22.014 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:37:22.014 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:37:22.014 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:37:22.014 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:37:22.014 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:37:22.014 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:37:22.014 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T23:37:22.014 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:37:22.014 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:37:22.014 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:37:22.014 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-25T23:37:22.015 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-25T23:37:22.015 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-25T23:37:22.015 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-25T23:37:22.015 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-25T23:37:22.015 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-25T23:37:22.015 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-25T23:37:22.237 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /elpers.sh:1669: wait_for_clean: eval 2024-03-25T23:37:22.237 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-25T23:37:22.237 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:37:22.237 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:37:22.238 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:37:22.238 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:37:22.238 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:37:22.238 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:37:22.238 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:37:22.238 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:37:22.238 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:37:22.238 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:37:22.238 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:37:22.238 INFO:tasks.workunit.client.0.smithi129.stderr:162264: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:37:22.238 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:37:22.238 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:37:22.238 INFO:tasks.workunit.client.0.smithi129.stderr:162264: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:37:22.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:37:22.239 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-25T23:37:22.239 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 162263 2024-03-25T23:37:23.158 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-25T23:37:23.158 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-25T23:37:23.158 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1 2024-03-25T23:37:23.158 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1 '!=' null 2024-03-25T23:37:23.158 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-03-25T23:37:23.158 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:37:23.158 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-25T23:37:23.158 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:37:23.158 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:37:23.158 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:37:23.158 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:37:23.159 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:37:23.159 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:37:23.159 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:37:23.159 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:37:23.159 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:37:23.159 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:37:23.159 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:37:23.159 INFO:tasks.workunit.client.0.smithi129.stderr:162266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:37:23.159 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:37:23.159 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:37:23.159 INFO:tasks.workunit.client.0.smithi129.stderr:162266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:37:23.159 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:37:23.159 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-25T23:37:23.159 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-25T23:37:23.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-25T23:37:23.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-03-25T23:37:23.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-25T23:37:23.160 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-03-25T23:37:23.161 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-25T23:37:23.161 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-25T23:37:23.161 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-25T23:37:23.161 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-25T23:37:23.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-03-25T23:37:23.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-03-25T23:37:23.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-25T23:37:23.493 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-25T23:37:23.494 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:37:23.494 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:37:23.494 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:37:23.494 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:37:23.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-25T23:36:51.008751+0000 2024-03-25T23:37:23.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-25T23:37:24.130 INFO:tasks.workunit.client.0.smithi129.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-25T23:37:24.147 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-25T23:36:51.008751+0000 2024-03-25T23:37:24.147 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-25T23:37:24.147 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-25T23:36:51.008751+0000 2024-03-25T23:37:24.147 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-25T23:37:24.147 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-25T23:37:24.147 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:37:24.147 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:37:24.148 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:37:24.148 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:37:24.148 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:37:24.149 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:37:24.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:36:51.008751+0000 '>' 2024-03-25T23:36:51.008751+0000 2024-03-25T23:37:24.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:37:25.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:37:25.462 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:37:25.462 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:37:25.462 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:37:25.463 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:37:25.463 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:37:25.463 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:37:25.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:36:51.008751+0000 '>' 2024-03-25T23:36:51.008751+0000 2024-03-25T23:37:25.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:37:26.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:37:26.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:37:26.783 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:37:26.783 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:37:26.783 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:37:26.784 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:37:26.784 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:37:27.092 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:36:51.008751+0000 '>' 2024-03-25T23:36:51.008751+0000 2024-03-25T23:37:27.092 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:37:28.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:37:28.094 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:37:28.094 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:37:28.095 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:37:28.095 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:37:28.095 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:37:28.095 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:37:28.404 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:36:51.008751+0000 '>' 2024-03-25T23:36:51.008751+0000 2024-03-25T23:37:28.404 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:37:29.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:37:29.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:37:29.407 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:37:29.407 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:37:29.407 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:37:29.408 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:37:29.408 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:37:29.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:36:51.008751+0000 '>' 2024-03-25T23:36:51.008751+0000 2024-03-25T23:37:29.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:37:30.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:37:30.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:37:30.731 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:37:30.731 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:37:30.731 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:37:30.732 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:37:30.732 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:37:31.041 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:37:24.248854+0000 '>' 2024-03-25T23:36:51.008751+0000 2024-03-25T23:37:31.041 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-25T23:37:31.041 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-03-25T23:37:31.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-25T23:37:31.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:37:31.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:37:31.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 165406"' 2024-03-25T23:37:31.042 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 165406' 2024-03-25T23:37:31.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-25T23:37:31.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:37:31.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:37:31.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-25T23:37:31.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/165408: /' 2024-03-25T23:37:31.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 165409"' 2024-03-25T23:37:31.042 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 165409' 2024-03-25T23:37:31.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-03-25T23:37:31.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 165406 165409' 2024-03-25T23:37:31.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-25T23:37:31.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-25T23:37:31.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 165406 2024-03-25T23:37:31.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-25T23:37:31.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/165412: /' 2024-03-25T23:37:31.985 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:37:31.985 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:37:31.985 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-25T23:37:31.985 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:37:31.985 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-25T23:37:31.985 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:37:31.985 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:37:31.985 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-25T23:37:31.985 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:37:31.986 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-25T23:37:31.986 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:37:31.986 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:37:31.986 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:37:31.986 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:37:31.986 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:37:31.986 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:37:31.986 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-25T23:37:31.986 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:37:31.986 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:37:31.987 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-25T23:37:31.987 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:37:31.987 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:37:31.987 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-25T23:37:31.987 INFO:tasks.workunit.client.0.smithi129.stderr:165412: _ 2024-03-25T23:37:31.987 INFO:tasks.workunit.client.0.smithi129.stderr:165412: hinfo_key 2024-03-25T23:37:31.987 INFO:tasks.workunit.client.0.smithi129.stderr:165412: snapset 2024-03-25T23:37:31.987 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-25T23:37:31.987 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:37:31.987 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:37:31.987 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-25T23:37:31.987 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:37:31.988 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:37:31.988 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:37:31.988 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:37:31.988 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helper165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:37:31.988 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:37:31.988 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-25T23:37:31.988 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:37:31.988 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-25T23:37:31.988 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:37:31.989 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:37:31.989 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-25T23:37:31.989 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:37:31.989 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-25T23:37:31.989 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:37:31.989 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:37:31.989 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:37:31.989 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:37:31.989 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:37:31.989 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:37:31.989 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-25T23:37:31.989 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:37:31.989 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:37:31.989 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-25T23:37:31.990 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:37:31.990 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-25T23:37:31.990 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-03-25T23:37:31.990 INFO:tasks.workunit.client.0.smithi129.stderr:165408: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-03-25T23:37:31.990 INFO:tasks.workunit.client.0.smithi129.stderr:165408: _ 2024-03-25T23:37:31.990 INFO:tasks.workunit.client.0.smithi129.stderr:165408: hinfo_key 2024-03-25T23:37:31.990 INFO:tasks.workunit.client.0.smithi129.stderr:165408: snapset 2024-03-25T23:37:31.990 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-25T23:37:31.990 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:37:31.990 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:37:31.990 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-25T23:37:31.991 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:37:31.991 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-25T23:37:31.991 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:37:31.992 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' s.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:37:31.992 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:37:31.992 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-25T23:37:31.992 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-25T23:37:31.992 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:37:31.992 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:37:31.992 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:37:31.992 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:37:31.993 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:37:31.993 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:37:31.993 INFO:tasks.workunit.client.0.smithi129.stderr:165412: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:37:31.993 INFO:tasks.workunit.client.0.smithi129.stderr:165412: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:37:31.993 INFO:tasks.workunit.client.0.smithi129.stderr:165412: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:37:31.993 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:37:31.993 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:37:31.993 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:37:31.993 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:37:31.993 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:37:31.993 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:37:31.993 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:37:31.993 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:37:31.994 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:37:31.994 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:37:31.994 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:37:31.994 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-25T23:37:31.994 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-25T23:37:31.994 INFO:tasks.workunit.client.0.smithi129.stderr:165412: start osd.1 2024-03-25T23:37:31.994 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec--osd-failsafe-full-ratio=.99' 2024-03-25T23:37:31.994 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:37:31.994 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:37:31.994 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-25T23:37:31.994 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-25T23:37:31.995 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:37:31.995 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:37:31.995 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:37:31.995 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:37:31.995 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:37:31.995 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:37:31.995 INFO:tasks.workunit.client.0.smithi129.stderr:165408: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:37:31.995 INFO:tasks.workunit.client.0.smithi129.stderr:165408: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:37:31.995 INFO:tasks.workunit.client.0.smithi129.stderr:165408: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:37:31.995 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:37:31.995 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:37:31.995 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:37:31.995 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:37:31.995 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:37:31.996 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:37:31.996 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:37:31.996 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:37:31.996 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:37:31.996 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:37:31.996 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:37:31.996 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-25T23:37:31.996 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-25T23:37:31.996 INFO:tasks.workunit.client.0.smithi129.stderr:165408: start osd.3 2024-03-25T23:37:36.410 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:37:36.410 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-25T23:37:36.411 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-25T23:37:36.411 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:37:36.411 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:37:36.411 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:37:36.411 INFO:tasks.workunit.client.0.smithi129.stderr:165408: 2024-03-25T23:37:32.045+0000 7f91f7d897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:37:36.411 INFO:tasks.workunit.client.0.smithi129.stderr:165408: 2024-03-25T23:37:32.045+0000 7f91f7d897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:37:36.411 INFO:tasks.workunit.client.0.smithi129.stderr:165408: 2024-03-25T23:37:32.049+0000 7f91f7d897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:37:36.411 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-25T23:37:36.411 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:37:36.411 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-25T23:37:36.411 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:37:36.411 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:37:36.411 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:37:36.411 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:37:36.412 INFO:tasks.workunit.client.0.smithi129.stderr:165408: 0 2024-03-25T23:37:36.412 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:37:36.412 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:37:36.412 INFO:tasks.workunit.client.0.smithi129.stderr:165408: 2024-03-25T23:37:32.621+0000 7f91f7d897c0 -1 Falling back to public interface 2024-03-25T23:37:36.412 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:37:36.412 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:37:36.412 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:37:36.412 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:37:36.412 INFO:tasks.workunit.client.0.smithi129.stderr:165408: 1 2024-03-25T23:37:36.412 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:37:36.412 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:37:36.412 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:37:36.412 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:37:36.412 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:37:36.413 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:37:36.413 INFO:tasks.workunit.client.0.smithi129.stderr:165408: 2 2024-03-25T23:37:36.413 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:37:36.413 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:37:36.413 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:37:36.413 INFO:tasks.workunit.client.0.smithi129.stderr:165408: 2024-03-25T23:37:36.369+0000 7f91f7d897c0 -1 osd.3 87 log_to_monitors true 2024-03-25T23:37:36.413 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:37:36.413 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:37:36.413 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:37:36.413 INFO:tasks.workunit.client.0.smithi129.stderr:165408: 3 2024-03-25T23:37:36.413 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:37:36.828 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperovery_ops 2024-03-25T23:37:36.828 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-25T23:37:36.829 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-25T23:37:36.829 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:37:36.829 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:37:36.829 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:37:36.829 INFO:tasks.workunit.client.0.smithi129.stderr:165412: 2024-03-25T23:37:32.021+0000 7efc99fd67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:37:36.829 INFO:tasks.workunit.client.0.smithi129.stderr:165412: 2024-03-25T23:37:32.025+0000 7efc99fd67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:37:36.829 INFO:tasks.workunit.client.0.smithi129.stderr:165412: 2024-03-25T23:37:32.025+0000 7efc99fd67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:37:36.829 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-25T23:37:36.829 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:37:36.829 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T23:37:36.829 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:37:36.829 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:37:36.830 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:37:36.830 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:37:36.830 INFO:tasks.workunit.client.0.smithi129.stderr:165412: 0 2024-03-25T23:37:36.830 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:37:36.830 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:37:36.830 INFO:tasks.workunit.client.0.smithi129.stderr:165412: 2024-03-25T23:37:32.601+0000 7efc99fd67c0 -1 Falling back to public interface 2024-03-25T23:37:36.830 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:37:36.830 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:37:36.830 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:37:36.830 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:37:36.830 INFO:tasks.workunit.client.0.smithi129.stderr:165412: 1 2024-03-25T23:37:36.831 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:37:36.831 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:37:36.831 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:37:36.831 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:37:36.831 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:37:36.831 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:37:36.831 INFO:tasks.workunit.client.0.smithi129.stderr:165412: 2 2024-03-25T23:37:36.831 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:37:36.831 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:37:36.831 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:37:36.831 INFO:tasks.workunit.client.0.smithi129.stderr:165412: 2024-03-25T23:37:36.093+0000 7efc99fd67c0 -1 osd.1 87 log_to_monitors true 2024-03-25T23:37:36.832 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:37:36.832 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:37:36.832 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:37:36.832 INFO:tasks.workunit.client.0.smithi129.stderr:165412: 3 2024-03-25T23:37:36.832 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:37:36.832 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:37:39.059 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/cs.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:37:39.059 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:37:39.059 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:37:39.059 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:37:39.059 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:37:39.059 INFO:tasks.workunit.client.0.smithi129.stderr:165408: 4 2024-03-25T23:37:39.059 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:37:39.059 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:37:39.059 INFO:tasks.workunit.client.0.smithi129.stderr:165408: osd.3 up in weight 1 up_from 93 up_thru 93 down_at 88 last_clean_interval [82,87) [v2:127.0.0.1:6826/3148679972,v1:127.0.0.1:6827/3148679972] [v2:127.0.0.1:6828/3148679972,v1:127.0.0.1:6829/3148679972] exists,up 614c4a39-b13c-4192-ba13-0218e9f2bcd7 2024-03-25T23:37:39.059 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:37:39.059 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:37:39.060 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:37:39.060 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:37:39.060 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:37:39.060 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:37:39.060 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:37:39.060 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:37:39.060 INFO:tasks.workunit.client.0.smithi129.stderr:165408: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:37:39.060 INFO:tasks.workunit.client.0.smithi129.stderr:165408: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:37:39.060 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:37:39.060 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:37:39.060 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:37:39.060 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:37:39.061 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:37:39.061 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:37:39.061 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:37:39.061 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:37:39.061 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:37:39.061 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:37:39.061 INFO:tasks.workunit.client.0.smithi129.stderr:165408: 1 2024-03-25T23:37:39.061 INFO:tasks.workunit.client.0.smithi129.stderr:165408: 2 2024-03-25T23:37:39.061 INFO:tasks.workunit.client.0.smithi129.stderr:165408: 3' 2024-03-25T23:37:39.061 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:37:39.061 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:37:39.061 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:37:39.061 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836549 2024-03-25T23:37:39.061 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836549 2024-03-25T23:37:39.062 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549' 2024-03-25T23:37:39.062 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperseph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:37:39.062 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:37:39.062 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:37:39.062 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:37:39.062 INFO:tasks.workunit.client.0.smithi129.stderr:165412: 4 2024-03-25T23:37:39.062 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:37:39.062 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:37:39.062 INFO:tasks.workunit.client.0.smithi129.stderr:165412: osd.1 up in weight 1 up_from 93 up_thru 85 down_at 88 last_clean_interval [85,87) [v2:127.0.0.1:6810/2009334143,v1:127.0.0.1:6812/2009334143] [v2:127.0.0.1:6814/2009334143,v1:127.0.0.1:6816/2009334143] exists,up 0a1d267b-774d-4fda-8cc0-7d86cd87d221 2024-03-25T23:37:39.062 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:37:39.062 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:37:39.062 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:37:39.062 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:37:39.062 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:37:39.063 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:37:39.063 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:37:39.063 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:37:39.063 INFO:tasks.workunit.client.0.smithi129.stderr:165412: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:37:39.063 INFO:tasks.workunit.client.0.smithi129.stderr:165412: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:37:39.063 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:37:39.063 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:37:39.063 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:37:39.063 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:37:39.063 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:37:39.063 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:37:39.063 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:37:39.063 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:37:39.063 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:37:39.063 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:37:39.064 INFO:tasks.workunit.client.0.smithi129.stderr:165412: 1 2024-03-25T23:37:39.064 INFO:tasks.workunit.client.0.smithi129.stderr:165412: 2 2024-03-25T23:37:39.064 INFO:tasks.workunit.client.0.smithi129.stderr:165412: 3' 2024-03-25T23:37:39.065 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:37:39.065 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:37:39.065 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:37:39.065 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836548 2024-03-25T23:37:39.065 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836548 2024-03-25T23:37:39.065 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548' 2024-03-25T23:37:39.065 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:37:41.207 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalon.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:37:41.207 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:37:41.207 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=399431958530 2024-03-25T23:37:41.207 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 399431958530 2024-03-25T23:37:41.208 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-399431958530' 2024-03-25T23:37:41.208 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:37:41.208 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:37:41.208 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579918 2024-03-25T23:37:41.208 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579918 2024-03-25T23:37:41.208 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-399431958530 2-317827579918' 2024-03-25T23:37:41.208 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:37:41.208 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:37:41.208 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=399431958531 2024-03-25T23:37:41.208 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 399431958531 2024-03-25T23:37:41.208 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-399431958530 2-317827579918 3-399431958531' 2024-03-25T23:37:41.208 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:37:41.208 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836549 2024-03-25T23:37:41.209 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:37:41.209 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:37:41.209 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836549 2024-03-25T23:37:41.209 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:37:41.209 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836549 2024-03-25T23:37:41.209 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836549' 2024-03-25T23:37:41.209 INFO:tasks.workunit.client.0.smithi129.stderr:165408: waiting osd.0 seq 21474836549 2024-03-25T23:37:41.209 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:37:41.209 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836547 -lt 21474836549 2024-03-25T23:37:41.209 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:37:41.209 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:37:41.209 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:37:41.209 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836549 -lt 21474836549 2024-03-25T23:37:41.209 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:37:41.210 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:37:41.210 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-399431958530 2024-03-25T23:37:41.211 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:37:41.212 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-399431958530 2024-03-25T23:37:41.218 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ube/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:37:41.219 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=399431958531 2024-03-25T23:37:41.219 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 399431958531 2024-03-25T23:37:41.219 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548 1-399431958531' 2024-03-25T23:37:41.219 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:37:41.219 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:37:41.219 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579917 2024-03-25T23:37:41.219 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579917 2024-03-25T23:37:41.219 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548 1-399431958531 2-317827579917' 2024-03-25T23:37:41.219 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:37:41.219 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:37:41.219 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=399431958530 2024-03-25T23:37:41.219 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 399431958530 2024-03-25T23:37:41.220 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548 1-399431958531 2-317827579917 3-399431958530' 2024-03-25T23:37:41.220 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:37:41.220 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836548 2024-03-25T23:37:41.220 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:37:41.220 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:37:41.220 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836548 2024-03-25T23:37:41.220 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:37:41.220 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836548 2024-03-25T23:37:41.220 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836548' 2024-03-25T23:37:41.220 INFO:tasks.workunit.client.0.smithi129.stderr:165412: waiting osd.0 seq 21474836548 2024-03-25T23:37:41.220 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:37:41.220 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836547 -lt 21474836548 2024-03-25T23:37:41.220 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:37:41.221 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:37:41.221 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:37:41.221 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836549 -lt 21474836548 2024-03-25T23:37:41.221 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:37:41.221 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-399431958531 2024-03-25T23:37:41.221 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:37:41.221 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:37:41.221 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-399431958531 2024-03-25T23:37:41.221 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:37:42.667 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=399431958531 2024-03-25T23:37:42.667 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 399431958531' 2024-03-25T23:37:42.667 INFO:tasks.workunit.client.0.smithi129.stderr:165412: waiting osd.1 seq 399431958531 2024-03-25T23:37:42.667 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:37:42.667 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958531 -lt 399431958531 2024-03-25T23:37:42.667 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:37:42.667 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-317827579917 2024-03-25T23:37:42.667 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:37:42.667 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:37:42.668 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-317827579917 2024-03-25T23:37:42.668 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:37:42.668 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579917 2024-03-25T23:37:42.668 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 317827579917' 2024-03-25T23:37:42.668 INFO:tasks.workunit.client.0.smithi129.stderr:165412: waiting osd.2 seq 317827579917 2024-03-25T23:37:42.668 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:37:42.668 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579919 -lt 317827579917 2024-03-25T23:37:42.668 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:37:42.668 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-399431958530 2024-03-25T23:37:42.668 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:37:42.668 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:37:42.668 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-399431958530 2024-03-25T23:37:42.668 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:37:42.669 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=399431958530 2024-03-25T23:37:42.669 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 399431958530' 2024-03-25T23:37:42.669 INFO:tasks.workunit.client.0.smithi129.stderr:165412: waiting osd.3 seq 399431958530 2024-03-25T23:37:42.669 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:37:42.669 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958531 -lt 399431958530 2024-03-25T23:37:42.669 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:37:42.669 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:37:42.669 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:37:42.669 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:37:42.669 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:37:42.669 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:37:42.669 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:37:42.669 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:37:42.676 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(untu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:37:42.676 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=399431958530 2024-03-25T23:37:42.676 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 399431958530' 2024-03-25T23:37:42.676 INFO:tasks.workunit.client.0.smithi129.stderr:165408: waiting osd.1 seq 399431958530 2024-03-25T23:37:42.676 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:37:42.677 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958531 -lt 399431958530 2024-03-25T23:37:42.677 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:37:42.677 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-317827579918 2024-03-25T23:37:42.677 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:37:42.677 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:37:42.677 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-317827579918 2024-03-25T23:37:42.677 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:37:42.677 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579918 2024-03-25T23:37:42.677 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 317827579918' 2024-03-25T23:37:42.677 INFO:tasks.workunit.client.0.smithi129.stderr:165408: waiting osd.2 seq 317827579918 2024-03-25T23:37:42.677 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:37:42.677 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579918 -lt 317827579918 2024-03-25T23:37:42.677 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:37:42.677 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-399431958531 2024-03-25T23:37:42.678 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:37:42.678 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:37:42.678 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-399431958531 2024-03-25T23:37:42.678 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:37:42.678 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=399431958531 2024-03-25T23:37:42.678 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 399431958531' 2024-03-25T23:37:42.678 INFO:tasks.workunit.client.0.smithi129.stderr:165408: waiting osd.3 seq 399431958531 2024-03-25T23:37:42.678 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:37:42.678 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958531 -lt 399431958531 2024-03-25T23:37:42.678 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:37:42.678 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:37:42.678 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:37:42.678 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:37:42.679 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:37:42.679 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:37:42.679 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:37:42.679 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:37:44.104 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cecontains("stale") | not)' 2024-03-25T23:37:44.104 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:37:44.104 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:37:44.104 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-25T23:37:44.104 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:37:44.104 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:37:44.104 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:37:44.104 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-25T23:37:44.105 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-25T23:37:44.105 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-25T23:37:44.105 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-25T23:37:44.105 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:37:44.105 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-25T23:37:44.105 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:37:44.105 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:37:44.105 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:37:44.105 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:37:44.105 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:37:44.105 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:37:44.105 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:37:44.105 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:37:44.105 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-25T23:37:44.106 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:37:44.106 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:37:44.106 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:37:44.106 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-25T23:37:44.106 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-25T23:37:44.106 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-25T23:37:44.106 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-25T23:37:44.106 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-25T23:37:44.106 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-25T23:37:44.112 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_secphtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:37:44.112 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:37:44.112 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:37:44.112 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-25T23:37:44.112 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:37:44.112 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:37:44.113 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:37:44.113 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-25T23:37:44.113 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-25T23:37:44.113 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-25T23:37:44.113 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-25T23:37:44.113 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:37:44.113 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-25T23:37:44.113 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:37:44.113 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:37:44.113 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:37:44.113 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:37:44.113 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:37:44.113 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:37:44.113 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:37:44.114 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:37:44.114 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-25T23:37:44.114 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:37:44.114 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:37:44.114 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:37:44.114 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-25T23:37:44.114 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-25T23:37:44.114 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-25T23:37:44.114 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-25T23:37:44.114 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-25T23:37:44.114 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-25T23:37:45.245 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/c 2024-03-25T23:37:45.246 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-25T23:37:45.246 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-25T23:37:45.246 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=2 2024-03-25T23:37:45.246 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 2 '!=' null 2024-03-25T23:37:45.246 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-03-25T23:37:45.246 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:37:45.246 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-25T23:37:45.246 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:37:45.246 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:37:45.246 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:37:45.246 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:37:45.246 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:37:45.246 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:37:45.247 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:37:45.247 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:37:45.247 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:37:45.247 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:37:45.247 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:37:45.247 INFO:tasks.workunit.client.0.smithi129.stderr:165412: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:37:45.247 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:37:45.247 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:37:45.247 INFO:tasks.workunit.client.0.smithi129.stderr:165412: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:37:45.247 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:37:45.253 INFO:tasks.workunit.client.0.smithi129.stderr:eph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-25T23:37:45.253 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-25T23:37:45.253 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-25T23:37:45.253 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=2 2024-03-25T23:37:45.254 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 2 '!=' null 2024-03-25T23:37:45.254 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-03-25T23:37:45.254 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:37:45.254 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-25T23:37:45.254 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:37:45.254 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:37:45.254 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:37:45.254 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:37:45.254 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:37:45.254 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:37:45.254 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:37:45.255 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:37:45.255 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:37:45.255 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:37:45.255 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:37:45.255 INFO:tasks.workunit.client.0.smithi129.stderr:165408: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:37:45.255 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:37:45.255 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:37:45.255 INFO:tasks.workunit.client.0.smithi129.stderr:165408: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:37:45.255 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:37:45.255 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-25T23:37:45.255 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 165409 2024-03-25T23:37:45.255 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-25T23:37:45.256 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-25T23:37:45.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-25T23:37:45.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-03-25T23:37:45.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-25T23:37:45.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-25T23:37:45.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-25T23:37:45.351 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-25T23:37:45.351 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-25T23:37:45.351 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T23:37:45.351 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-25T23:37:45.352 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:37:45.352 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:37:45.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:37:45.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:37:45.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:37:45.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:37:45.461 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T23:37:45.462 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T23:37:45.462 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T23:37:45.463 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T23:37:45.463 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T23:37:45.464 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T23:37:45.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:37:45.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T23:37:45.465 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T23:37:45.465 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:37:45.466 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T23:37:45.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T23:37:45.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-25T23:37:45.496 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T23:37:45.497 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:37:45.497 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:37:45.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-25T23:37:45.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T23:37:45.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T23:37:45.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-25T23:37:45.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-25T23:37:45.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-25T23:37:45.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-25T23:37:45.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-25T23:37:45.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T23:37:45.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-25T23:37:45.499 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:37:45.499 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:37:45.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:37:45.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:37:45.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:37:45.501 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:37:45.501 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T23:37:45.502 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T23:37:45.503 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T23:37:45.503 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T23:37:45.503 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T23:37:45.504 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T23:37:45.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:37:45.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T23:37:45.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T23:37:45.505 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:37:45.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T23:37:45.506 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T23:37:45.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-25T23:37:45.508 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T23:37:45.508 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:37:45.508 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:37:45.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-25T23:37:45.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T23:37:45.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T23:37:45.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-25T23:37:45.510 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-25T23:37:45.510 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:37:45.510 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:37:45.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61636 2024-03-25T23:37:45.511 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-25T23:37:45.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-25T23:37:45.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-25T23:37:45.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-25T23:37:45.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_jerasure_overwrites td/osd-scrub-repair 2024-03-25T23:37:45.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_corrupt_and_repair_jerasure_overwrites: '[' true = true ']' 2024-03-25T23:37:45.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:820: TEST_corrupt_and_repair_jerasure_overwrites: corrupt_and_repair_jerasure td/osd-scrub-repair true 2024-03-25T23:37:45.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:798: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2024-03-25T23:37:45.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:799: corrupt_and_repair_jerasure: local allow_overwrites=true 2024-03-25T23:37:45.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:800: corrupt_and_repair_jerasure: local poolname=ecpool 2024-03-25T23:37:45.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:802: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2024-03-25T23:37:45.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-25T23:37:45.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-25T23:37:45.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-25T23:37:45.514 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-25T23:37:45.514 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-25T23:37:45.514 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-25T23:37:45.962 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-25T23:37:45.962 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:37:45.962 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:37:45.962 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:37:45.962 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:37:45.962 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:37:45.962 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:37:45.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-25T23:37:46.181 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-25T23:37:46.183 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-25T23:37:46.183 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T23:37:46.183 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T23:37:46.183 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-25T23:37:46.183 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-25T23:37:46.184 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T23:37:46.184 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T23:37:46.184 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T23:37:46.185 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T23:37:46.186 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:37:46.186 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:37:46.186 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-25T23:37:46.186 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T23:37:46.186 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get fsid 2024-03-25T23:37:46.298 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-25T23:37:46.298 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T23:37:46.298 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T23:37:46.298 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-25T23:37:46.298 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-25T23:37:46.300 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T23:37:46.300 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T23:37:46.300 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T23:37:46.300 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T23:37:46.300 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:37:46.300 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:37:46.300 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-25T23:37:46.300 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T23:37:46.301 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get mon_host 2024-03-25T23:37:46.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:803: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2024-03-25T23:37:46.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-25T23:37:46.407 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-25T23:37:46.407 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-25T23:37:46.407 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-25T23:37:46.407 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-25T23:37:46.407 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-25T23:37:47.283 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-25T23:37:47.284 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:37:47.284 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:37:47.284 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:37:47.284 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:37:47.284 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:37:47.285 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:37:47.286 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T23:37:47.287 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T23:37:47.323 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: seq 0 3 2024-03-25T23:37:47.324 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-03-25T23:37:47.324 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 0 2024-03-25T23:37:47.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:37:47.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:37:47.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-25T23:37:47.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:37:47.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-25T23:37:47.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:37:47.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:37:47.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:37:47.327 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:37:47.327 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-25T23:37:47.327 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-25T23:37:47.327 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:37:47.327 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:37:47.327 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:37:47.327 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:37:47.327 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:37:47.327 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:37:47.327 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:37:47.327 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:37:47.327 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:37:47.327 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:37:47.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:37:47.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:37:47.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:37:47.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:37:47.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:37:47.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:37:47.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:37:47.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:37:47.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:37:47.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:37:47.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:37:47.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T23:37:47.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-25T23:37:47.330 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:37:47.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9df5d5cf-07d3-4b0f-a72f-f18c27e925f4 2024-03-25T23:37:47.331 INFO:tasks.workunit.client.0.smithi129.stdout:add osd0 9df5d5cf-07d3-4b0f-a72f-f18c27e925f4 2024-03-25T23:37:47.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9df5d5cf-07d3-4b0f-a72f-f18c27e925f4' 2024-03-25T23:37:47.331 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:37:47.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDLCgJmdqjfFBAA4SYjxsf7kvYUZfLmuZlTDw== 2024-03-25T23:37:47.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDLCgJmdqjfFBAA4SYjxsf7kvYUZfLmuZlTDw=="}' 2024-03-25T23:37:47.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9df5d5cf-07d3-4b0f-a72f-f18c27e925f4 -i td/osd-scrub-repair/0/new.json 2024-03-25T23:37:47.610 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:37:47.628 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-25T23:37:47.629 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDLCgJmdqjfFBAA4SYjxsf7kvYUZfLmuZlTDw== --osd-uuid 9df5d5cf-07d3-4b0f-a72f-f18c27e925f4 2024-03-25T23:37:47.662 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:37:47.661+0000 7f885efd97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:37:47.677 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:37:47.677+0000 7f885efd97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:37:47.678 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:37:47.681+0000 7f885efd97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:37:47.679 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:37:47.681+0000 7f885efd97c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-25T23:37:47.679 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:37:47.681+0000 7f885efd97c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-25T23:37:47.679 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:37:47.681+0000 7f885efd97c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-25T23:37:47.769 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:37:47.769+0000 7f885efd97c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-25T23:37:54.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-25T23:37:54.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:37:54.126 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd0 key to auth repository 2024-03-25T23:37:54.126 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-25T23:37:54.126 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:37:54.520 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.0 2024-03-25T23:37:54.520 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-25T23:37:54.520 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:37:54.521 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:37:54.521 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:37:54.521 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:37:54.554 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:37:54.553+0000 7f9e1ac1f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:37:54.577 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:37:54.577+0000 7f9e1ac1f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:37:54.579 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:37:54.581+0000 7f9e1ac1f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:37:54.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-25T23:37:54.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:37:54.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T23:37:54.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:37:54.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:37:54.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:37:54.864 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:37:54.864 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:37:54.864 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:37:54.864 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:37:55.133 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:37:55.133+0000 7f9e1ac1f7c0 -1 Falling back to public interface 2024-03-25T23:37:55.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:37:56.174 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:37:56.174 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:37:56.174 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:37:56.174 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:37:56.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:37:56.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:37:56.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:37:57.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:37:57.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:37:57.495 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:37:57.495 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:37:57.495 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:37:57.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:37:57.653 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:37:57.653+0000 7f9e1ac1f7c0 -1 osd.0 0 log_to_monitors true 2024-03-25T23:37:57.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:37:58.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:37:58.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:37:58.835 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:37:58.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:37:58.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:37:58.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:37:59.222 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:38:00.218 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:38:00.218 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:38:00.219 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:38:00.219 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:38:00.219 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:38:00.219 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:38:00.663 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:38:01.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:38:01.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:38:01.665 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:38:01.666 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:38:01.666 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:38:01.666 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:38:01.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:38:02.989 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:38:02.990 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:38:02.990 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:38:02.990 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:38:02.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:38:02.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:38:03.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:38:04.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:38:04.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:38:04.309 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:38:04.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:38:04.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:38:04.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:38:04.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:38:05.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:38:05.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:38:05.615 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:38:05.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:38:05.616 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:38:05.616 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:38:05.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:38:06.924 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:38:06.924 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:38:06.924 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:38:06.925 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:38:06.925 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:38:06.925 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:38:07.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:38:08.268 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:38:08.268 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:38:08.269 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:38:08.269 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:38:08.269 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:38:08.269 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:38:08.590 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:38:09.593 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:38:09.593 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:38:09.594 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:38:09.594 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:38:09.595 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:38:09.595 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:38:09.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:38:10.909 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:38:10.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:38:10.910 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:38:10.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:38:10.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:38:10.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:38:11.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:38:12.321 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:38:12.321 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:38:12.322 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:38:12.323 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:38:12.323 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:38:12.323 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:38:13.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:38:14.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:38:14.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:38:14.189 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:38:14.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:38:14.190 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:38:14.190 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:38:15.075 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:38:16.076 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:38:16.077 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:38:16.077 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:38:16.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:38:16.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:38:16.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:38:16.980 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:38:17.982 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:38:17.982 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:38:17.983 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:38:17.983 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:38:17.983 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:38:17.983 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:38:18.411 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:38:19.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:38:19.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:38:19.414 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:38:19.414 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:38:19.414 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:38:19.414 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:38:20.223 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:38:21.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:38:21.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:38:21.226 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:38:21.226 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:38:21.226 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:38:21.226 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:38:21.551 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:38:22.552 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:38:22.552 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:38:22.552 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:38:22.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:38:22.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:38:22.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:38:22.878 INFO:tasks.workunit.client.0.smithi129.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1912183245,v1:127.0.0.1:6803/1912183245] [v2:127.0.0.1:6804/1912183245,v1:127.0.0.1:6805/1912183245] exists,up 9df5d5cf-07d3-4b0f-a72f-f18c27e925f4 2024-03-25T23:38:22.879 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:38:22.880 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:38:22.880 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:38:22.880 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-03-25T23:38:22.880 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 1 2024-03-25T23:38:22.880 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:38:22.880 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:38:22.880 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-25T23:38:22.880 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:38:22.880 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:38:22.880 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:38:22.880 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:38:22.880 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:38:22.880 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:38:22.880 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-25T23:38:22.881 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-25T23:38:22.881 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:38:22.881 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:38:22.881 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:38:22.883 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:38:22.883 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:38:22.883 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:38:22.883 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:38:22.883 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:38:22.883 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:38:22.883 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:38:22.883 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:38:22.883 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:38:22.883 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:38:22.883 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:38:22.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:38:22.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:38:22.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:38:22.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:38:22.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:38:22.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:38:22.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:38:22.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T23:38:22.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-25T23:38:22.884 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:38:22.885 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e1891468-d0f5-4d04-9e87-8f7a4ef38ae8 2024-03-25T23:38:22.885 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 e1891468-d0f5-4d04-9e87-8f7a4ef38ae8' 2024-03-25T23:38:22.885 INFO:tasks.workunit.client.0.smithi129.stdout:add osd1 e1891468-d0f5-4d04-9e87-8f7a4ef38ae8 2024-03-25T23:38:22.885 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:38:22.905 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDuCgJmyM8bNhAAY+MaTYjyzyBYmoXmZYHzsQ== 2024-03-25T23:38:22.905 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDuCgJmyM8bNhAAY+MaTYjyzyBYmoXmZYHzsQ=="}' 2024-03-25T23:38:22.906 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e1891468-d0f5-4d04-9e87-8f7a4ef38ae8 -i td/osd-scrub-repair/1/new.json 2024-03-25T23:38:23.311 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:38:23.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-25T23:38:23.333 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDuCgJmyM8bNhAAY+MaTYjyzyBYmoXmZYHzsQ== --osd-uuid e1891468-d0f5-4d04-9e87-8f7a4ef38ae8 2024-03-25T23:38:23.365 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:38:23.364+0000 7f545100c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:38:23.413 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:38:23.412+0000 7f545100c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:38:23.415 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:38:23.416+0000 7f545100c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:38:23.416 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:38:23.416+0000 7f545100c7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-25T23:38:23.416 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:38:23.416+0000 7f545100c7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-25T23:38:23.416 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:38:23.416+0000 7f545100c7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-25T23:38:23.496 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:38:23.496+0000 7f545100c7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-25T23:38:30.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-25T23:38:30.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:38:30.150 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd1 key to auth repository 2024-03-25T23:38:30.151 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-25T23:38:30.151 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:38:30.535 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.1 2024-03-25T23:38:30.536 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-25T23:38:30.536 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:38:30.536 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:38:30.536 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:38:30.536 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:38:30.568 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:38:30.568+0000 7f376f7b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:38:30.584 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:38:30.584+0000 7f376f7b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:38:30.586 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:38:30.584+0000 7f376f7b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:38:30.864 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-25T23:38:30.864 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:38:30.864 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:38:30.865 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T23:38:30.865 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:38:30.865 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:38:30.865 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:38:30.865 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:38:30.865 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:38:30.865 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:38:31.148 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:38:31.148+0000 7f376f7b37c0 -1 Falling back to public interface 2024-03-25T23:38:31.216 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:38:32.218 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:38:32.218 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:38:32.218 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:38:32.219 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:38:32.219 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:38:32.219 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:38:32.522 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:38:33.524 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:38:33.524 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:38:33.525 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:38:33.525 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:38:33.525 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:38:33.526 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:38:33.720 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:38:33.720+0000 7f376f7b37c0 -1 osd.1 0 log_to_monitors true 2024-03-25T23:38:33.849 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:38:34.850 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:38:34.851 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:38:34.851 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:38:34.851 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:38:34.852 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:38:34.852 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:38:35.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:38:36.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:38:36.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:38:36.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:38:36.167 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:38:36.167 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:38:36.167 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:38:36.482 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:38:37.482 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:38:37.482 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:38:37.483 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:38:37.483 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:38:37.483 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:38:37.483 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:38:37.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:38:38.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:38:38.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:38:38.832 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:38:38.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:38:38.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:38:38.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:38:39.147 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:38:40.149 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:38:40.149 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:38:40.149 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:38:40.149 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:38:40.150 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:38:40.150 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:38:40.469 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:38:41.470 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:38:41.471 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:38:41.471 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:38:41.471 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:38:41.471 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:38:41.472 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:38:41.849 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:38:42.851 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:38:42.851 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:38:42.851 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:38:42.852 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:38:42.852 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:38:42.852 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:38:43.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:38:44.171 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:38:44.171 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:38:44.171 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:38:44.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:38:44.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:38:44.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:38:44.478 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:38:45.480 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:38:45.480 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:38:45.481 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:38:45.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:38:45.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:38:45.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:38:45.816 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:38:46.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:38:46.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:38:46.818 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:38:46.818 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:38:46.819 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:38:46.819 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:38:47.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:38:48.143 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:38:48.143 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:38:48.144 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:38:48.144 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:38:48.144 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:38:48.144 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:38:48.937 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:38:49.939 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:38:49.939 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:38:49.940 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:38:49.940 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:38:49.940 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:38:49.940 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:38:50.763 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:38:51.765 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:38:51.765 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:38:51.766 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:38:51.766 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:38:51.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:38:51.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:38:52.427 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:38:53.427 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:38:53.427 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:38:53.427 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:38:53.428 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:38:53.428 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:38:53.428 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:38:53.999 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:38:55.001 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:38:55.001 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:38:55.002 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:38:55.002 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:38:55.002 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:38:55.002 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:38:55.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:38:56.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:38:56.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:38:56.734 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:38:56.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:38:56.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:38:56.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:38:57.049 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:38:58.051 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:38:58.052 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:38:58.052 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:38:58.052 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:38:58.052 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:38:58.053 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:38:58.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:38:59.787 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:38:59.787 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:38:59.788 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-25T23:38:59.788 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-25T23:38:59.788 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:38:59.788 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:39:00.175 INFO:tasks.workunit.client.0.smithi129.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/935306853,v1:127.0.0.1:6811/935306853] [v2:127.0.0.1:6812/935306853,v1:127.0.0.1:6813/935306853] exists,up e1891468-d0f5-4d04-9e87-8f7a4ef38ae8 2024-03-25T23:39:00.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:39:00.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:39:00.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:39:00.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-03-25T23:39:00.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 2 2024-03-25T23:39:00.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:39:00.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:39:00.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-25T23:39:00.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:39:00.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-25T23:39:00.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:39:00.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:39:00.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:39:00.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:39:00.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-25T23:39:00.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-25T23:39:00.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:39:00.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:39:00.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:39:00.178 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:39:00.178 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:39:00.178 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:39:00.178 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:39:00.178 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:39:00.178 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:39:00.179 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:39:00.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:39:00.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:39:00.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:39:00.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:39:00.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:39:00.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:39:00.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:39:00.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:39:00.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:39:00.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:39:00.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:39:00.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T23:39:00.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-25T23:39:00.181 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:39:00.181 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=51e83ed2-2c31-4ea0-b586-33c43ca2595f 2024-03-25T23:39:00.181 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 51e83ed2-2c31-4ea0-b586-33c43ca2595f' 2024-03-25T23:39:00.181 INFO:tasks.workunit.client.0.smithi129.stdout:add osd2 51e83ed2-2c31-4ea0-b586-33c43ca2595f 2024-03-25T23:39:00.181 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:39:00.202 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAUCwJmKG0tDBAAZJdT0Jbo7tm4to6WEJfULQ== 2024-03-25T23:39:00.202 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAUCwJmKG0tDBAAZJdT0Jbo7tm4to6WEJfULQ=="}' 2024-03-25T23:39:00.202 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 51e83ed2-2c31-4ea0-b586-33c43ca2595f -i td/osd-scrub-repair/2/new.json 2024-03-25T23:39:00.544 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:39:00.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-25T23:39:00.563 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAUCwJmKG0tDBAAZJdT0Jbo7tm4to6WEJfULQ== --osd-uuid 51e83ed2-2c31-4ea0-b586-33c43ca2595f 2024-03-25T23:39:00.597 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:39:00.595+0000 7fa6f622d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:39:00.628 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:39:00.627+0000 7fa6f622d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:39:00.630 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:39:00.627+0000 7fa6f622d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:39:00.631 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:39:00.627+0000 7fa6f622d7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-25T23:39:00.631 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:39:00.627+0000 7fa6f622d7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-25T23:39:00.631 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:39:00.627+0000 7fa6f622d7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-25T23:39:00.699 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:39:00.699+0000 7fa6f622d7c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-25T23:39:07.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-25T23:39:07.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:39:07.491 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd2 key to auth repository 2024-03-25T23:39:07.491 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-25T23:39:07.491 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:39:07.910 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.2 2024-03-25T23:39:07.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-25T23:39:07.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:39:07.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:39:07.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:39:07.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:39:07.941 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:39:07.939+0000 7f191f0a37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:39:07.968 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:39:07.967+0000 7f191f0a37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:39:07.969 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:39:07.967+0000 7f191f0a37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:39:08.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-25T23:39:08.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:39:08.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-25T23:39:08.237 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:39:08.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:39:08.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:39:08.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:39:08.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:39:08.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:39:08.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:39:08.537 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:39:08.535+0000 7f191f0a37c0 -1 Falling back to public interface 2024-03-25T23:39:08.569 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:39:09.570 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:39:09.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:39:09.918 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:39:09.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:39:09.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:39:09.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:39:09.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:39:10.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:39:10.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:39:10.874 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:39:10.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:39:10.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:39:10.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:39:11.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:39:11.204 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:39:11.203+0000 7f191f0a37c0 -1 osd.2 0 log_to_monitors true 2024-03-25T23:39:12.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:39:12.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:39:12.177 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:39:12.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:39:12.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:39:12.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:39:12.604 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:39:13.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:39:13.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:39:13.607 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:39:13.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:39:13.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:39:13.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:39:13.934 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:39:14.936 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:39:14.936 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:39:14.936 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:39:14.937 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:39:14.937 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:39:14.937 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:39:15.351 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:39:16.353 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:39:16.353 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:39:16.353 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:39:16.353 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:39:16.354 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:39:16.354 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:39:16.748 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:39:17.750 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:39:17.751 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:39:17.751 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:39:17.751 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:39:17.751 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:39:17.751 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:39:18.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:39:19.209 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:39:19.209 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:39:19.210 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:39:19.211 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:39:19.211 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:39:19.211 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:39:19.531 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:39:20.531 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:39:20.531 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:39:20.532 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:39:20.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:39:20.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:39:20.533 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:39:20.921 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:39:21.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:39:21.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:39:21.923 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:39:21.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:39:21.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:39:21.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:39:22.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:39:23.462 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:39:23.462 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:39:23.462 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:39:23.463 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:39:23.463 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:39:23.463 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:39:23.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:39:24.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:39:24.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:39:24.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:39:24.783 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:39:24.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:39:24.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:39:25.192 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:39:26.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:39:26.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:39:26.194 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:39:26.195 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:39:26.195 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:39:26.195 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:39:27.203 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:39:28.204 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:39:28.204 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:39:28.205 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:39:28.205 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:39:28.205 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:39:28.205 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:39:28.559 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:39:29.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:39:29.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:39:29.561 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:39:29.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:39:29.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:39:29.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:39:30.053 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:39:31.055 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:39:31.055 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:39:31.055 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:39:31.056 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:39:31.056 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:39:31.056 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:39:31.628 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:39:32.629 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:39:32.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:39:32.630 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:39:32.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:39:32.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:39:32.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:39:33.308 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:39:34.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:39:34.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:39:34.311 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:39:34.312 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:39:34.312 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:39:34.312 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:39:34.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:39:35.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:39:35.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:39:35.962 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:39:35.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:39:35.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:39:35.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:39:36.543 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:39:37.545 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:39:37.545 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:39:37.545 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-25T23:39:37.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-25T23:39:37.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:39:37.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:39:37.865 INFO:tasks.workunit.client.0.smithi129.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1848320480,v1:127.0.0.1:6819/1848320480] [v2:127.0.0.1:6820/1848320480,v1:127.0.0.1:6821/1848320480] exists,up 51e83ed2-2c31-4ea0-b586-33c43ca2595f 2024-03-25T23:39:37.867 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:39:37.867 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:39:37.867 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:39:37.867 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-03-25T23:39:37.867 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 3 2024-03-25T23:39:37.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:39:37.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:39:37.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-03-25T23:39:37.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:39:37.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-03-25T23:39:37.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:39:37.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:39:37.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:39:37.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:39:37.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-25T23:39:37.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-25T23:39:37.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:39:37.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:39:37.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:39:37.869 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:39:37.869 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:39:37.869 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:39:37.869 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:39:37.869 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:39:37.869 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:39:37.869 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:39:37.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:39:37.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:39:37.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:39:37.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:39:37.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:39:37.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:39:37.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:39:37.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:39:37.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:39:37.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:39:37.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:39:37.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T23:39:37.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-03-25T23:39:37.871 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:39:37.871 INFO:tasks.workunit.client.0.smithi129.stdout:add osd3 2de53dc4-8cd5-4295-9d23-e6a36c158ac1 2024-03-25T23:39:37.871 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2de53dc4-8cd5-4295-9d23-e6a36c158ac1 2024-03-25T23:39:37.871 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 2de53dc4-8cd5-4295-9d23-e6a36c158ac1' 2024-03-25T23:39:37.872 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:39:37.893 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA5CwJmExlVNRAAoTvLta+MLWeTTlZoaex3Tg== 2024-03-25T23:39:37.893 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA5CwJmExlVNRAAoTvLta+MLWeTTlZoaex3Tg=="}' 2024-03-25T23:39:37.893 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2de53dc4-8cd5-4295-9d23-e6a36c158ac1 -i td/osd-scrub-repair/3/new.json 2024-03-25T23:39:38.447 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:39:38.465 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-03-25T23:39:38.466 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA5CwJmExlVNRAAoTvLta+MLWeTTlZoaex3Tg== --osd-uuid 2de53dc4-8cd5-4295-9d23-e6a36c158ac1 2024-03-25T23:39:38.501 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:39:38.498+0000 7f3453cf57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:39:38.548 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:39:38.546+0000 7f3453cf57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:39:38.551 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:39:38.550+0000 7f3453cf57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:39:38.552 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:39:38.550+0000 7f3453cf57c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-25T23:39:38.552 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:39:38.550+0000 7f3453cf57c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-25T23:39:38.552 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:39:38.550+0000 7f3453cf57c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-25T23:39:38.636 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:39:38.634+0000 7f3453cf57c0 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-03-25T23:39:45.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-03-25T23:39:45.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:39:45.187 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-03-25T23:39:45.188 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd3 key to auth repository 2024-03-25T23:39:45.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:39:45.577 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.3 2024-03-25T23:39:45.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-03-25T23:39:45.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:39:45.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:39:45.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:39:45.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:39:45.609 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:39:45.606+0000 7f574978a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:39:45.611 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:39:45.610+0000 7f574978a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:39:45.612 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:39:45.610+0000 7f574978a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:39:45.913 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-03-25T23:39:45.913 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:39:45.913 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-25T23:39:45.913 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:39:45.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:39:45.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:39:45.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:39:45.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:39:45.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:39:45.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:39:46.161 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:39:46.158+0000 7f574978a7c0 -1 Falling back to public interface 2024-03-25T23:39:46.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:39:47.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:39:47.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:39:47.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:39:47.256 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:39:47.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:39:47.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:39:47.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:39:48.566 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:39:48.566 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:39:48.566 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:39:48.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:39:48.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:39:48.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:39:48.852 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:39:48.850+0000 7f574978a7c0 -1 osd.3 0 log_to_monitors true 2024-03-25T23:39:48.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:39:49.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:39:49.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:39:49.874 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:39:49.875 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:39:49.875 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:39:49.875 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:39:50.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:39:51.181 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:39:51.181 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:39:51.181 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:39:51.182 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:39:51.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:39:51.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:39:51.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:39:52.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:39:52.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:39:52.495 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:39:52.495 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:39:52.495 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:39:52.495 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:39:52.961 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:39:53.965 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:39:53.965 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:39:53.966 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:39:53.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:39:53.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:39:53.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:39:54.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:39:55.277 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:39:55.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:39:55.278 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:39:55.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:39:55.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:39:55.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:39:55.597 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:39:56.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:39:56.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:39:56.599 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:39:56.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:39:56.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:39:56.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:39:56.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:39:57.909 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:39:57.909 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:39:57.909 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:39:57.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:39:57.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:39:57.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:39:58.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:39:59.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:39:59.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:39:59.230 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:39:59.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:39:59.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:39:59.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:39:59.576 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:40:00.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:40:00.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:40:00.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:40:00.579 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:40:00.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:40:00.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:40:01.001 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:40:02.003 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:40:02.003 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:40:02.003 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:40:02.004 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:40:02.004 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:40:02.004 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:40:02.317 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:40:03.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:40:03.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:40:03.320 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:40:03.320 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:40:03.320 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:40:03.320 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:40:03.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:40:04.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:40:04.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:40:04.648 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:40:04.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:40:04.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:40:04.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:40:05.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:40:06.417 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:40:06.417 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:40:06.417 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:40:06.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:40:06.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:40:06.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:40:07.216 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:40:08.218 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:40:08.218 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:40:08.218 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:40:08.219 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:40:08.219 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:40:08.219 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:40:09.109 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:40:10.111 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:40:10.111 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:40:10.112 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:40:10.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:40:10.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:40:10.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:40:10.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:40:11.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:40:11.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:40:11.873 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:40:11.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:40:11.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:40:11.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:40:12.357 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:40:13.358 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:40:13.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:40:13.359 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:40:13.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:40:13.367 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:40:13.368 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:40:14.084 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:40:15.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:40:15.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:40:15.086 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-25T23:40:15.087 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-25T23:40:15.087 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:40:15.087 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:40:15.428 INFO:tasks.workunit.client.0.smithi129.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2450363022,v1:127.0.0.1:6827/2450363022] [v2:127.0.0.1:6828/2450363022,v1:127.0.0.1:6829/2450363022] exists,up 2de53dc4-8cd5-4295-9d23-e6a36c158ac1 2024-03-25T23:40:15.429 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:40:15.429 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:40:15.429 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:40:15.429 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:807: corrupt_and_repair_jerasure: create_rbd_pool 2024-03-25T23:40:15.429 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-25T23:40:15.762 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'rbd' does not exist 2024-03-25T23:40:15.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-25T23:40:15.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-25T23:40:16.257 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'rbd' created 2024-03-25T23:40:16.274 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T23:40:17.275 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-25T23:40:20.215 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: corrupt_and_repair_jerasure: wait_for_clean 2024-03-25T23:40:20.215 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:40:20.216 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:40:20.216 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:40:20.216 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:40:20.217 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:40:20.217 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:40:20.218 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:40:20.218 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:40:20.218 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:40:20.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:40:20.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:40:20.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:40:20.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:40:20.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:40:20.345 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:40:20.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:40:20.665 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:40:20.665 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:40:20.665 INFO:tasks.workunit.client.0.smithi129.stderr:3' 2024-03-25T23:40:20.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:40:20.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:40:20.666 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:40:20.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836507 2024-03-25T23:40:20.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836507 2024-03-25T23:40:20.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507' 2024-03-25T23:40:20.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:40:20.831 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:40:20.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672983 2024-03-25T23:40:20.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672983 2024-03-25T23:40:20.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-42949672983' 2024-03-25T23:40:20.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:40:20.973 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:40:21.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509462 2024-03-25T23:40:21.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509462 2024-03-25T23:40:21.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-42949672983 2-64424509462' 2024-03-25T23:40:21.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:40:21.113 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:40:21.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345932 2024-03-25T23:40:21.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345932 2024-03-25T23:40:21.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-42949672983 2-64424509462 3-85899345932' 2024-03-25T23:40:21.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:40:21.256 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836507 2024-03-25T23:40:21.257 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:40:21.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:40:21.258 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836507 2024-03-25T23:40:21.258 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:40:21.259 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 21474836507 2024-03-25T23:40:21.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836507 2024-03-25T23:40:21.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-03-25T23:40:21.260 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:40:21.558 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836507 2024-03-25T23:40:21.558 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:40:22.559 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:40:22.560 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:40:22.871 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836507 -lt 21474836507 2024-03-25T23:40:22.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:40:22.873 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672983 2024-03-25T23:40:22.873 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:40:22.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:40:22.875 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672983 2024-03-25T23:40:22.875 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:40:22.876 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672983 2024-03-25T23:40:22.876 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 42949672983 2024-03-25T23:40:22.877 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-03-25T23:40:22.877 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:40:23.241 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672983 -lt 42949672983 2024-03-25T23:40:23.241 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:40:23.242 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509462 2024-03-25T23:40:23.242 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:40:23.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:40:23.244 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509462 2024-03-25T23:40:23.244 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:40:23.245 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509462 2024-03-25T23:40:23.246 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 64424509462 2024-03-25T23:40:23.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-03-25T23:40:23.246 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:40:23.635 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509462 -lt 64424509462 2024-03-25T23:40:23.635 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:40:23.636 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345932 2024-03-25T23:40:23.636 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:40:23.638 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:40:23.639 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345932 2024-03-25T23:40:23.639 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:40:23.640 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345932 2024-03-25T23:40:23.640 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345932' 2024-03-25T23:40:23.641 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.3 seq 85899345932 2024-03-25T23:40:23.641 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:40:23.965 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345932 -lt 85899345932 2024-03-25T23:40:23.966 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:40:23.967 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:40:23.967 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:40:24.445 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-25T23:40:24.445 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:40:24.446 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:40:24.446 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:40:24.446 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:40:24.446 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:40:24.446 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:40:24.446 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:40:24.786 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T23:40:24.786 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:40:24.787 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:40:24.787 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:40:25.161 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-25T23:40:25.161 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:40:25.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:40:25.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:810: corrupt_and_repair_jerasure: create_ec_pool ecpool true k=2 m=2 2024-03-25T23:40:25.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-25T23:40:25.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-25T23:40:25.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-03-25T23:40:25.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-25T23:40:25.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-03-25T23:40:25.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-25T23:40:25.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-25T23:40:26.526 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'ecpool' created 2024-03-25T23:40:26.547 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T23:40:27.549 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-03-25T23:40:27.549 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-03-25T23:40:28.109 INFO:tasks.workunit.client.0.smithi129.stderr:set pool 2 allow_ec_overwrites to true 2024-03-25T23:40:28.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-25T23:40:28.133 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:40:28.133 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:40:28.133 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:40:28.133 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:40:28.134 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:40:28.134 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:40:28.134 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:40:28.134 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:40:28.134 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:40:28.250 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:40:28.250 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:40:28.250 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:40:28.250 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:40:28.250 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:40:28.250 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:40:28.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:40:28.608 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:40:28.608 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:40:28.608 INFO:tasks.workunit.client.0.smithi129.stderr:3' 2024-03-25T23:40:28.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:40:28.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:40:28.609 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:40:28.818 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836510 2024-03-25T23:40:28.818 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836510 2024-03-25T23:40:28.818 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510' 2024-03-25T23:40:28.819 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:40:28.819 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:40:28.961 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672985 2024-03-25T23:40:28.961 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672985 2024-03-25T23:40:28.961 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510 1-42949672985' 2024-03-25T23:40:28.961 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:40:28.962 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:40:29.103 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509465 2024-03-25T23:40:29.103 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509465 2024-03-25T23:40:29.103 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510 1-42949672985 2-64424509465' 2024-03-25T23:40:29.103 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:40:29.104 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:40:29.244 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345934 2024-03-25T23:40:29.244 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345934 2024-03-25T23:40:29.244 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510 1-42949672985 2-64424509465 3-85899345934' 2024-03-25T23:40:29.244 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:40:29.245 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836510 2024-03-25T23:40:29.245 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:40:29.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:40:29.247 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836510 2024-03-25T23:40:29.247 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:40:29.247 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 21474836510 2024-03-25T23:40:29.248 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836510 2024-03-25T23:40:29.248 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-03-25T23:40:29.248 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:40:29.540 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836510 2024-03-25T23:40:29.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:40:30.542 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:40:30.543 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:40:30.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836510 -lt 21474836510 2024-03-25T23:40:30.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:40:30.861 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672985 2024-03-25T23:40:30.861 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:40:30.862 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:40:30.863 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672985 2024-03-25T23:40:30.863 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:40:30.864 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672985 2024-03-25T23:40:30.865 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 42949672985 2024-03-25T23:40:30.865 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-03-25T23:40:30.865 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:40:31.192 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672986 -lt 42949672985 2024-03-25T23:40:31.193 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:40:31.193 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509465 2024-03-25T23:40:31.193 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:40:31.195 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:40:31.196 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509465 2024-03-25T23:40:31.196 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:40:31.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509465 2024-03-25T23:40:31.197 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 64424509465 2024-03-25T23:40:31.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509465' 2024-03-25T23:40:31.198 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:40:31.518 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509465 -lt 64424509465 2024-03-25T23:40:31.519 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:40:31.519 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:40:31.520 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345934 2024-03-25T23:40:31.520 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:40:31.521 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345934 2024-03-25T23:40:31.522 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:40:31.524 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.3 seq 85899345934 2024-03-25T23:40:31.524 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345934 2024-03-25T23:40:31.524 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345934' 2024-03-25T23:40:31.524 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:40:31.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345934 -lt 85899345934 2024-03-25T23:40:31.839 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:40:31.839 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:40:31.839 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:40:32.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:40:32.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:40:32.284 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:40:32.285 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:40:32.285 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:40:32.285 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:40:32.285 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:40:32.285 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:40:32.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:40:32.601 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:40:32.601 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:40:32.601 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:40:32.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:40:32.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:40:32.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:40:32.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-25T23:40:32.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:811: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-03-25T23:40:32.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-03-25T23:40:32.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-03-25T23:40:32.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-03-25T23:40:32.965 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-25T23:40:32.965 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-25T23:40:32.965 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-03-25T23:40:32.965 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-25T23:40:32.965 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-25T23:40:32.965 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-25T23:40:33.419 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-25T23:40:33.438 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-25T23:40:33.846 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-25T23:40:33.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-25T23:40:33.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-25T23:40:33.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-25T23:40:34.179 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-03-25T23:40:34.179 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-03-25T23:40:34.179 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-25T23:40:34.179 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-03-25T23:40:34.179 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-25T23:40:34.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-03-25T23:40:34.508 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-03-25T23:40:34.508 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-03-25T23:40:34.508 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-25T23:40:34.508 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-03-25T23:40:34.508 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-03-25T23:40:34.508 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T23:40:34.846 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-25T23:40:34.846 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:40:34.846 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:40:34.846 INFO:tasks.workunit.client.0.smithi129.stderr:0' 2024-03-25T23:40:34.846 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 0 2024-03-25T23:40:34.846 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('1' '2' '0') 2024-03-25T23:40:34.846 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-03-25T23:40:34.846 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=1 2024-03-25T23:40:34.847 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=2 2024-03-25T23:40:34.847 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-03-25T23:40:34.847 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-25T23:40:34.847 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-03-25T23:40:34.847 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-03-25T23:40:34.847 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-25T23:40:34.847 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:40:34.847 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:40:34.847 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-25T23:40:34.847 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:40:34.847 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-03-25T23:40:34.847 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:40:34.847 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:40:34.847 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-25T23:40:34.848 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:40:34.848 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-25T23:40:34.848 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:40:34.848 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:40:34.848 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:40:34.848 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:40:34.848 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:40:36.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:40:36.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-03-25T23:40:36.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:40:36.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:40:36.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-25T23:40:36.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:40:36.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-25T23:40:36.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-03-25T23:40:37.765 INFO:tasks.workunit.client.0.smithi129.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-03-25T23:40:38.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-25T23:40:38.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:40:38.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:40:38.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-25T23:40:38.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:40:38.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-25T23:40:38.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:40:38.311 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:40:38.311 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:40:38.311 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:40:38.311 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-25T23:40:38.311 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-25T23:40:38.311 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:40:38.311 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:40:38.311 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:40:38.317 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:40:38.317 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:40:38.317 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:40:38.317 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:40:38.317 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:40:38.318 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:40:38.318 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:40:38.318 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:40:38.318 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:40:38.318 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:40:38.318 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:40:38.318 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:40:38.318 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:40:38.318 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:40:38.318 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:40:38.318 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:40:38.318 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:40:38.318 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-25T23:40:38.318 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-25T23:40:38.319 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.3 2024-03-25T23:40:38.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:40:38.319 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-25T23:40:38.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-25T23:40:38.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:40:38.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:40:38.320 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:40:38.348 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:40:38.349+0000 7fb6eaa827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:40:38.350 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:40:38.349+0000 7fb6eaa827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:40:38.351 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:40:38.353+0000 7fb6eaa827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:40:38.638 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-25T23:40:38.639 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:40:38.639 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-25T23:40:38.639 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:40:38.640 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:40:38.640 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:40:38.640 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:40:38.640 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-25T23:40:38.640 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:40:38.641 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:40:38.893 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:40:38.893+0000 7fb6eaa827c0 -1 Falling back to public interface 2024-03-25T23:40:38.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:40:39.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:40:39.974 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:40:39.974 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:40:39.974 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:40:39.974 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:40:39.974 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:40:40.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:40:41.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:40:41.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:40:41.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:40:41.280 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:40:41.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:40:41.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:40:41.583 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:40:41.585+0000 7fb6eaa827c0 -1 osd.3 38 log_to_monitors true 2024-03-25T23:40:41.594 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:40:42.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:40:42.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:40:42.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:40:42.596 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:40:42.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:40:42.597 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:40:42.951 INFO:tasks.workunit.client.0.smithi129.stderr:osd.3 up in weight 1 up_from 43 up_thru 43 down_at 39 last_clean_interval [20,38) [v2:127.0.0.1:6826/2583895246,v1:127.0.0.1:6827/2583895246] [v2:127.0.0.1:6828/2583895246,v1:127.0.0.1:6829/2583895246] exists,up 2de53dc4-8cd5-4295-9d23-e6a36c158ac1 2024-03-25T23:40:42.951 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:40:42.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:40:42.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:40:42.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:40:42.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:40:42.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:40:42.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:40:42.952 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:40:42.953 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:40:42.953 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:40:42.953 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:40:42.953 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:40:42.953 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:40:43.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:40:43.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:40:43.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:40:43.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:40:43.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:40:43.062 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:40:43.699 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:40:43.699 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:40:43.699 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:40:43.699 INFO:tasks.workunit.client.0.smithi129.stderr:3' 2024-03-25T23:40:43.699 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:40:43.700 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:40:43.700 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:40:43.925 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836514 2024-03-25T23:40:43.925 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836514 2024-03-25T23:40:43.925 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514' 2024-03-25T23:40:43.925 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:40:43.926 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:40:44.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672989 2024-03-25T23:40:44.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672989 2024-03-25T23:40:44.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514 1-42949672989' 2024-03-25T23:40:44.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:40:44.069 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:40:44.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509469 2024-03-25T23:40:44.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509469 2024-03-25T23:40:44.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514 1-42949672989 2-64424509469' 2024-03-25T23:40:44.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:40:44.348 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:40:44.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=184683593730 2024-03-25T23:40:44.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 184683593730 2024-03-25T23:40:44.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514 1-42949672989 2-64424509469 3-184683593730' 2024-03-25T23:40:44.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:40:44.494 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836514 2024-03-25T23:40:44.494 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:40:44.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:40:44.495 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836514 2024-03-25T23:40:44.495 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:40:44.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836514 2024-03-25T23:40:44.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836514' 2024-03-25T23:40:44.496 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 21474836514 2024-03-25T23:40:44.497 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:40:44.808 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836513 -lt 21474836514 2024-03-25T23:40:44.808 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:40:45.809 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:40:45.810 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:40:46.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836514 -lt 21474836514 2024-03-25T23:40:46.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:40:46.125 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672989 2024-03-25T23:40:46.126 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:40:46.127 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:40:46.128 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672989 2024-03-25T23:40:46.128 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:40:46.129 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672989 2024-03-25T23:40:46.129 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672989' 2024-03-25T23:40:46.130 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 42949672989 2024-03-25T23:40:46.130 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:40:46.447 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672990 -lt 42949672989 2024-03-25T23:40:46.448 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:40:46.448 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509469 2024-03-25T23:40:46.449 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:40:46.451 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:40:46.452 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509469 2024-03-25T23:40:46.452 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:40:46.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509469 2024-03-25T23:40:46.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509469' 2024-03-25T23:40:46.453 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 64424509469 2024-03-25T23:40:46.453 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:40:46.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509469 -lt 64424509469 2024-03-25T23:40:46.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:40:46.783 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-184683593730 2024-03-25T23:40:46.783 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:40:46.785 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:40:46.786 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-184683593730 2024-03-25T23:40:46.786 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:40:46.787 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=184683593730 2024-03-25T23:40:46.787 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 184683593730' 2024-03-25T23:40:46.788 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.3 seq 184683593730 2024-03-25T23:40:46.788 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:40:47.100 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 184683593730 -lt 184683593730 2024-03-25T23:40:47.101 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:40:47.101 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:40:47.102 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:40:47.495 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:40:47.495 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:40:47.496 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:40:47.496 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:40:47.496 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:40:47.496 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:40:47.496 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:40:47.496 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:40:47.813 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T23:40:47.814 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:40:47.814 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:40:47.814 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:40:48.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-25T23:40:48.190 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-25T23:40:48.190 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-25T23:40:48.190 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-25T23:40:48.190 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:40:48.190 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-25T23:40:48.291 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:40:48.291 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:40:48.292 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:40:48.292 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:40:48.292 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:40:48.292 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:40:48.292 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:40:48.292 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:40:48.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T23:40:48.588 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:40:48.588 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:40:48.588 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:40:48.941 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-25T23:40:48.941 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-25T23:40:48.941 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-25T23:40:48.941 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-25T23:40:48.942 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-25T23:40:48.942 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-25T23:40:48.942 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-25T23:40:48.942 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-25T23:40:48.942 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-25T23:40:49.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-25T23:40:49.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-25T23:40:49.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-25T23:40:49.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:40:49.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-25T23:40:49.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:40:49.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:40:49.486 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:40:49.486 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:40:49.486 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:40:49.486 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:40:49.486 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:40:49.487 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:40:49.799 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T23:40:49.799 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:40:49.800 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:40:49.800 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:40:50.164 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-25T23:40:50.164 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-25T23:40:50.164 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-25T23:40:50.164 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-25T23:40:50.164 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-25T23:40:50.164 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-25T23:40:50.164 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-25T23:40:50.165 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-25T23:40:50.165 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-25T23:40:50.595 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-25T23:40:50.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-25T23:40:50.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-25T23:40:50.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:40:50.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-25T23:40:50.996 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:40:50.997 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:40:50.997 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:40:50.997 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:40:50.998 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:40:50.998 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:40:50.998 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:40:50.998 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:40:51.314 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:40:51.315 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:40:51.316 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:40:51.316 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:40:51.685 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:40:51.685 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:40:51.685 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:40:51.685 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-03-25T23:40:51.685 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-25T23:40:51.685 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-25T23:40:51.686 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-25T23:40:51.686 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-25T23:40:52.006 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-03-25T23:40:52.007 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-03-25T23:40:52.007 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-25T23:40:52.007 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-25T23:40:52.007 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:40:52.008 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:40:52.008 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:40:52.008 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:40:52.318 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-25T23:40:26.411152+0000 2024-03-25T23:40:52.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-25T23:40:52.619 INFO:tasks.workunit.client.0.smithi129.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-25T23:40:52.638 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-25T23:40:26.411152+0000 2024-03-25T23:40:52.638 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-25T23:40:52.638 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-25T23:40:26.411152+0000 2024-03-25T23:40:52.638 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-25T23:40:52.638 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-25T23:40:52.638 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:40:52.638 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:40:52.638 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:40:52.639 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:40:52.639 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:40:52.639 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:40:52.945 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:40:26.411152+0000 '>' 2024-03-25T23:40:26.411152+0000 2024-03-25T23:40:52.945 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:40:53.946 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:40:53.946 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:40:53.947 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:40:53.947 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:40:53.947 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:40:53.948 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:40:53.948 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:40:54.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:40:26.411152+0000 '>' 2024-03-25T23:40:26.411152+0000 2024-03-25T23:40:54.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:40:55.266 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:40:55.266 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:40:55.267 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:40:55.267 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:40:55.267 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:40:55.268 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:40:55.268 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:40:55.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:40:26.411152+0000 '>' 2024-03-25T23:40:26.411152+0000 2024-03-25T23:40:55.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:40:56.589 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:40:56.589 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:40:56.590 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:40:56.590 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:40:56.590 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:40:56.590 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:40:56.590 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:40:56.905 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:40:26.411152+0000 '>' 2024-03-25T23:40:26.411152+0000 2024-03-25T23:40:56.905 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:40:57.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:40:57.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:40:57.908 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:40:57.908 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:40:57.908 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:40:57.908 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:40:57.909 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:40:58.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:40:52.778766+0000 '>' 2024-03-25T23:40:26.411152+0000 2024-03-25T23:40:58.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-25T23:40:58.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-25T23:40:58.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:40:58.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:40:58.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-25T23:40:58.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:40:58.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-25T23:40:58.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:40:58.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:40:58.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-25T23:40:58.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:40:58.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-25T23:40:58.232 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:40:58.232 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:40:58.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:40:58.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:40:58.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:40:58.542 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:40:58.542 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-25T23:40:58.542 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:40:58.542 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:40:58.542 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-25T23:40:58.542 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:40:58.542 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-25T23:40:58.542 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-03-25T23:40:58.861 INFO:tasks.workunit.client.0.smithi129.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-03-25T23:40:58.861 INFO:tasks.workunit.client.0.smithi129.stdout:_ 2024-03-25T23:40:58.861 INFO:tasks.workunit.client.0.smithi129.stdout:hinfo_key 2024-03-25T23:40:58.861 INFO:tasks.workunit.client.0.smithi129.stdout:snapset 2024-03-25T23:40:59.150 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-25T23:40:59.151 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:40:59.151 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:40:59.151 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-25T23:40:59.151 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:40:59.151 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-25T23:40:59.151 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:40:59.151 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:40:59.151 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:40:59.151 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:40:59.151 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-25T23:40:59.151 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-25T23:40:59.152 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:40:59.152 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:40:59.152 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:40:59.152 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:40:59.152 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:40:59.152 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:40:59.154 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:40:59.154 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:40:59.154 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:40:59.154 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:40:59.154 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:40:59.154 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:40:59.154 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:40:59.154 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:40:59.154 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:40:59.154 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:40:59.154 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:40:59.155 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:40:59.155 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:40:59.155 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:40:59.155 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-25T23:40:59.155 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-25T23:40:59.155 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.3 2024-03-25T23:40:59.156 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:40:59.156 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-25T23:40:59.156 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-25T23:40:59.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:40:59.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:40:59.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:40:59.195 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:40:59.196+0000 7ff9b05aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:40:59.253 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:40:59.252+0000 7ff9b05aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:40:59.254 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:40:59.256+0000 7ff9b05aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:40:59.488 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-25T23:40:59.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:40:59.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-25T23:40:59.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:40:59.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:40:59.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:40:59.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:40:59.489 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-25T23:40:59.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:40:59.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:40:59.791 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:40:59.788+0000 7ff9b05aa7c0 -1 Falling back to public interface 2024-03-25T23:41:00.020 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:41:01.022 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:41:01.022 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:41:01.022 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:41:01.022 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:41:01.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:41:01.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:41:01.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:41:02.338 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:41:02.338 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:41:02.339 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:41:02.339 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:41:02.339 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:41:02.339 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:41:02.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:41:03.040 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:41:03.040+0000 7ff9b05aa7c0 -1 osd.3 45 log_to_monitors true 2024-03-25T23:41:03.678 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:41:03.678 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:41:03.678 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:41:03.678 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:41:03.678 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:41:03.679 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:41:04.245 INFO:tasks.workunit.client.0.smithi129.stderr:osd.3 up in weight 1 up_from 50 up_thru 50 down_at 46 last_clean_interval [43,45) [v2:127.0.0.1:6826/280435430,v1:127.0.0.1:6827/280435430] [v2:127.0.0.1:6828/280435430,v1:127.0.0.1:6829/280435430] exists,up 2de53dc4-8cd5-4295-9d23-e6a36c158ac1 2024-03-25T23:41:04.245 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:41:04.245 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:41:04.245 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:41:04.245 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:41:04.245 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:41:04.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:41:04.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:41:04.246 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:41:04.246 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:41:04.247 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:41:04.247 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:41:04.247 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:41:04.247 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:41:04.364 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:41:04.364 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:41:04.364 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:41:04.364 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:41:04.364 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:41:04.364 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:41:04.719 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:41:04.719 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:41:04.719 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:41:04.719 INFO:tasks.workunit.client.0.smithi129.stderr:3' 2024-03-25T23:41:04.719 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:41:04.719 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:41:04.719 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:41:04.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836519 2024-03-25T23:41:04.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836519 2024-03-25T23:41:04.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836519' 2024-03-25T23:41:04.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:41:04.870 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:41:05.054 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672995 2024-03-25T23:41:05.054 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672995 2024-03-25T23:41:05.054 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836519 1-42949672995' 2024-03-25T23:41:05.054 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:41:05.055 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:41:05.204 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509474 2024-03-25T23:41:05.204 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509474 2024-03-25T23:41:05.204 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836519 1-42949672995 2-64424509474' 2024-03-25T23:41:05.204 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:41:05.204 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:41:05.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364802 2024-03-25T23:41:05.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364802 2024-03-25T23:41:05.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836519 1-42949672995 2-64424509474 3-214748364802' 2024-03-25T23:41:05.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:41:05.350 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836519 2024-03-25T23:41:05.350 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:41:05.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:41:05.351 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836519 2024-03-25T23:41:05.351 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:41:05.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836519 2024-03-25T23:41:05.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836519' 2024-03-25T23:41:05.352 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 21474836519 2024-03-25T23:41:05.352 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:41:05.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836518 -lt 21474836519 2024-03-25T23:41:05.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:41:06.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:41:06.651 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:41:06.977 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836519 -lt 21474836519 2024-03-25T23:41:06.977 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:41:06.978 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672995 2024-03-25T23:41:06.978 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:41:06.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:41:06.980 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672995 2024-03-25T23:41:06.980 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:41:06.981 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672995 2024-03-25T23:41:06.982 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672995' 2024-03-25T23:41:06.982 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 42949672995 2024-03-25T23:41:06.982 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:41:07.301 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672995 -lt 42949672995 2024-03-25T23:41:07.302 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:41:07.302 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509474 2024-03-25T23:41:07.302 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:41:07.304 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:41:07.304 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509474 2024-03-25T23:41:07.305 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:41:07.306 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509474 2024-03-25T23:41:07.306 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509474' 2024-03-25T23:41:07.306 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 64424509474 2024-03-25T23:41:07.306 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:41:07.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509474 -lt 64424509474 2024-03-25T23:41:07.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:41:07.655 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:41:07.655 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-214748364802 2024-03-25T23:41:07.656 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:41:07.657 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-214748364802 2024-03-25T23:41:07.657 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:41:07.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364802 2024-03-25T23:41:07.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 214748364802' 2024-03-25T23:41:07.660 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.3 seq 214748364802 2024-03-25T23:41:07.660 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:41:07.976 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364802 -lt 214748364802 2024-03-25T23:41:07.977 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:41:07.978 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:41:07.978 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:41:08.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:41:08.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:41:08.343 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:41:08.343 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:41:08.344 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:41:08.344 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:41:08.344 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:41:08.344 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:41:08.643 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T23:41:08.643 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:41:08.643 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:41:08.644 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:41:09.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-25T23:41:09.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-25T23:41:09.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-25T23:41:09.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-25T23:41:09.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:41:09.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-25T23:41:09.122 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:41:09.122 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:41:09.123 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:41:09.123 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:41:09.123 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:41:09.123 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:41:09.124 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:41:09.124 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:41:09.439 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T23:41:09.439 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:41:09.440 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:41:09.440 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:41:09.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-25T23:41:09.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-25T23:41:09.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-25T23:41:09.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-25T23:41:09.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-25T23:41:09.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-25T23:41:09.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-25T23:41:09.806 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-25T23:41:09.806 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-25T23:41:10.223 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-25T23:41:10.223 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-25T23:41:10.223 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-25T23:41:10.223 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:41:10.223 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-25T23:41:10.424 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:41:10.424 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:41:10.425 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:41:10.425 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:41:10.425 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:41:10.425 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:41:10.425 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:41:10.426 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:41:10.718 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:41:10.718 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:41:10.718 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:41:10.718 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:41:11.092 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:41:11.092 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:41:11.092 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:41:11.092 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-25T23:41:11.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-25T23:41:11.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2024-03-25T23:41:11.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-25T23:41:11.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-03-25T23:41:11.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-03-25T23:41:11.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-25T23:41:11.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:41:11.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:41:11.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-25T23:41:11.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:41:11.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-25T23:41:11.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:41:11.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:41:11.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-25T23:41:11.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:41:11.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-25T23:41:11.139 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:41:11.139 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:41:11.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:41:11.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:41:11.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:41:11.445 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:41:11.445 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-25T23:41:11.445 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:41:11.445 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:41:11.445 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-25T23:41:11.445 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:41:11.445 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:41:11.445 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-25T23:41:13.480 INFO:tasks.workunit.client.0.smithi129.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-03-25T23:41:14.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-25T23:41:14.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:41:14.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:41:14.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-25T23:41:14.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:41:14.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:41:14.018 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:41:14.018 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:41:14.018 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:41:14.018 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:41:14.018 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-25T23:41:14.018 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-25T23:41:14.018 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:41:14.018 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:41:14.018 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:41:14.018 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:41:14.018 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:41:14.018 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:41:14.020 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:41:14.020 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:41:14.021 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:41:14.021 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:41:14.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:41:14.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:41:14.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:41:14.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:41:14.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:41:14.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:41:14.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:41:14.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:41:14.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:41:14.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:41:14.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-25T23:41:14.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-25T23:41:14.022 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.1 2024-03-25T23:41:14.022 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:41:14.025 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-25T23:41:14.025 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-25T23:41:14.025 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:41:14.025 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:41:14.026 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:41:14.059 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:41:14.060+0000 7f02fbcd77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:41:14.062 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:41:14.060+0000 7f02fbcd77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:41:14.063 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:41:14.064+0000 7f02fbcd77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:41:14.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-25T23:41:14.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:41:14.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T23:41:14.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:41:14.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:41:14.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:41:14.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:41:14.344 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-25T23:41:14.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:41:14.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:41:14.596 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:41:14.596+0000 7f02fbcd77c0 -1 Falling back to public interface 2024-03-25T23:41:14.673 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:41:15.675 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:41:15.675 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:41:15.675 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:41:15.675 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:41:15.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:41:15.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:41:15.995 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:41:16.999 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:41:16.999 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:41:16.999 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:41:16.999 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:41:16.999 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:41:16.999 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:41:17.296 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:41:17.296+0000 7f02fbcd77c0 -1 osd.1 52 log_to_monitors true 2024-03-25T23:41:17.305 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:41:18.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:41:18.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:41:18.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:41:18.307 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:41:18.308 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:41:18.308 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:41:18.706 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1 up in weight 1 up_from 57 up_thru 57 down_at 53 last_clean_interval [10,52) [v2:127.0.0.1:6810/4113951872,v1:127.0.0.1:6811/4113951872] [v2:127.0.0.1:6812/4113951872,v1:127.0.0.1:6813/4113951872] exists,up e1891468-d0f5-4d04-9e87-8f7a4ef38ae8 2024-03-25T23:41:18.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:41:18.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:41:18.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:41:18.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:41:18.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:41:18.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:41:18.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:41:18.707 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:41:18.707 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:41:18.707 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:41:18.707 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:41:18.708 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:41:18.708 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:41:18.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:41:18.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:41:18.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:41:18.811 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:41:18.811 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:41:18.811 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:41:19.323 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:41:19.324 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:41:19.324 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:41:19.324 INFO:tasks.workunit.client.0.smithi129.stderr:3' 2024-03-25T23:41:19.324 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:41:19.324 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:41:19.324 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:41:19.469 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836523 2024-03-25T23:41:19.469 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836523 2024-03-25T23:41:19.469 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836523' 2024-03-25T23:41:19.469 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:41:19.470 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:41:19.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135874 2024-03-25T23:41:19.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135874 2024-03-25T23:41:19.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836523 1-244813135874' 2024-03-25T23:41:19.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:41:19.713 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:41:19.856 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509478 2024-03-25T23:41:19.856 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509478 2024-03-25T23:41:19.856 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836523 1-244813135874 2-64424509478' 2024-03-25T23:41:19.856 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:41:19.856 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:41:20.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364806 2024-03-25T23:41:20.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364806 2024-03-25T23:41:20.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836523 1-244813135874 2-64424509478 3-214748364806' 2024-03-25T23:41:20.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:41:20.023 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836523 2024-03-25T23:41:20.023 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:41:20.024 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:41:20.025 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836523 2024-03-25T23:41:20.025 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:41:20.026 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836523 2024-03-25T23:41:20.026 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836523' 2024-03-25T23:41:20.026 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 21474836523 2024-03-25T23:41:20.027 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:41:20.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836523 -lt 21474836523 2024-03-25T23:41:20.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:41:20.346 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-244813135874 2024-03-25T23:41:20.346 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:41:20.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:41:20.348 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-244813135874 2024-03-25T23:41:20.349 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:41:20.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135874 2024-03-25T23:41:20.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 244813135874' 2024-03-25T23:41:20.350 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 244813135874 2024-03-25T23:41:20.350 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:41:20.674 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135874 -lt 244813135874 2024-03-25T23:41:20.674 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:41:20.675 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509478 2024-03-25T23:41:20.675 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:41:20.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:41:20.677 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:41:20.677 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509478 2024-03-25T23:41:20.678 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509478 2024-03-25T23:41:20.678 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509478' 2024-03-25T23:41:20.678 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 64424509478 2024-03-25T23:41:20.679 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:41:20.995 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509477 -lt 64424509478 2024-03-25T23:41:20.996 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:41:21.997 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:41:21.998 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:41:22.301 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509478 -lt 64424509478 2024-03-25T23:41:22.301 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:41:22.302 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-214748364806 2024-03-25T23:41:22.302 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:41:22.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:41:22.304 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-214748364806 2024-03-25T23:41:22.304 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:41:22.305 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364806 2024-03-25T23:41:22.305 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 214748364806' 2024-03-25T23:41:22.306 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.3 seq 214748364806 2024-03-25T23:41:22.306 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:41:22.643 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364806 -lt 214748364806 2024-03-25T23:41:22.643 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:41:22.644 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:41:22.644 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:41:23.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:41:23.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:41:23.036 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:41:23.037 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:41:23.037 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:41:23.037 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:41:23.037 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:41:23.037 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:41:23.353 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T23:41:23.354 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:41:23.354 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:41:23.355 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:41:23.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-25T23:41:23.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-25T23:41:23.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-25T23:41:23.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-25T23:41:23.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:41:23.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-25T23:41:23.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:41:23.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:41:23.806 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:41:23.806 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:41:23.806 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:41:23.806 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:41:23.807 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:41:23.807 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:41:24.114 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:41:24.115 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:41:24.115 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:41:24.116 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:41:24.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:41:24.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:41:24.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:41:24.528 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-03-25T23:41:24.528 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-25T23:41:24.528 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-25T23:41:24.528 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-25T23:41:24.528 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-25T23:41:24.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-03-25T23:41:24.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-03-25T23:41:24.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-25T23:41:24.840 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-25T23:41:24.841 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:41:24.841 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:41:24.841 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:41:24.841 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:41:25.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-25T23:40:52.778766+0000 2024-03-25T23:41:25.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-25T23:41:25.423 INFO:tasks.workunit.client.0.smithi129.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-25T23:41:25.441 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-25T23:40:52.778766+0000 2024-03-25T23:41:25.441 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-25T23:41:25.441 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-25T23:40:52.778766+0000 2024-03-25T23:41:25.442 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-25T23:41:25.442 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-25T23:41:25.442 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:41:25.442 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:41:25.442 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:41:25.442 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:41:25.443 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:41:25.443 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:41:25.750 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:40:52.778766+0000 '>' 2024-03-25T23:40:52.778766+0000 2024-03-25T23:41:25.750 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:41:26.751 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:41:26.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:41:26.753 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:41:26.753 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:41:26.753 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:41:26.753 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:41:26.753 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:41:27.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:40:52.778766+0000 '>' 2024-03-25T23:40:52.778766+0000 2024-03-25T23:41:27.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:41:28.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:41:28.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:41:28.070 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:41:28.070 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:41:28.071 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:41:28.071 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:41:28.071 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:41:28.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:40:52.778766+0000 '>' 2024-03-25T23:40:52.778766+0000 2024-03-25T23:41:28.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:41:29.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:41:29.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:41:29.389 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:41:29.389 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:41:29.389 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:41:29.389 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:41:29.390 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:41:29.719 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:40:52.778766+0000 '>' 2024-03-25T23:40:52.778766+0000 2024-03-25T23:41:29.719 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:41:30.720 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:41:30.720 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:41:30.721 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:41:30.722 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:41:30.722 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:41:30.722 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:41:30.722 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:41:31.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:41:25.991748+0000 '>' 2024-03-25T23:40:52.778766+0000 2024-03-25T23:41:31.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-25T23:41:31.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-25T23:41:31.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:41:31.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:41:31.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-25T23:41:31.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:41:31.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-25T23:41:31.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:41:31.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:41:31.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-25T23:41:31.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:41:31.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-25T23:41:31.033 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:41:31.033 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:41:31.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:41:31.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:41:31.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:41:31.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:41:31.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-25T23:41:31.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:41:31.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:41:31.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-25T23:41:31.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:41:31.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-25T23:41:31.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-03-25T23:41:31.614 INFO:tasks.workunit.client.0.smithi129.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-03-25T23:41:31.615 INFO:tasks.workunit.client.0.smithi129.stdout:_ 2024-03-25T23:41:31.615 INFO:tasks.workunit.client.0.smithi129.stdout:hinfo_key 2024-03-25T23:41:31.615 INFO:tasks.workunit.client.0.smithi129.stdout:snapset 2024-03-25T23:41:31.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-25T23:41:31.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:41:31.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:41:31.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-25T23:41:31.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:41:31.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-25T23:41:31.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:41:31.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:41:31.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:41:31.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:41:31.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-25T23:41:31.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-25T23:41:31.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:41:31.924 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:41:31.924 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:41:31.924 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:41:31.924 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:41:31.924 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:41:31.926 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:41:31.926 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:41:31.926 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:41:31.926 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:41:31.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:41:31.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:41:31.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:41:31.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:41:31.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:41:31.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:41:31.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:41:31.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:41:31.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:41:31.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:41:31.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-25T23:41:31.928 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-25T23:41:31.928 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.3 2024-03-25T23:41:31.928 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:41:31.928 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-25T23:41:31.928 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-25T23:41:31.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:41:31.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:41:31.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:41:31.959 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:41:31.960+0000 7f540e9ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:41:31.961 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:41:31.960+0000 7f540e9ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:41:31.963 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:41:31.964+0000 7f540e9ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:41:32.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-25T23:41:32.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:41:32.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-25T23:41:32.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:41:32.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:41:32.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:41:32.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:41:32.344 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-25T23:41:32.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:41:32.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:41:32.512 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:41:32.512+0000 7f540e9ed7c0 -1 Falling back to public interface 2024-03-25T23:41:32.672 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:41:33.674 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:41:33.674 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:41:33.674 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:41:33.674 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:41:33.675 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:41:33.675 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:41:34.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:41:35.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:41:35.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:41:35.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:41:35.017 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:41:35.018 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:41:35.018 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:41:35.340 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:41:35.464 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:41:35.464+0000 7f540e9ed7c0 -1 osd.3 59 log_to_monitors true 2024-03-25T23:41:36.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:41:36.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:41:36.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:41:36.341 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:41:36.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:41:36.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:41:36.669 INFO:tasks.workunit.client.0.smithi129.stderr:osd.3 up in weight 1 up_from 64 up_thru 64 down_at 60 last_clean_interval [50,59) [v2:127.0.0.1:6826/659008993,v1:127.0.0.1:6827/659008993] [v2:127.0.0.1:6828/659008993,v1:127.0.0.1:6829/659008993] exists,up 2de53dc4-8cd5-4295-9d23-e6a36c158ac1 2024-03-25T23:41:36.669 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:41:36.669 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:41:36.669 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:41:36.669 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:41:36.669 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:41:36.669 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:41:36.669 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:41:36.670 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:41:36.670 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:41:36.670 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:41:36.670 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:41:36.670 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:41:36.670 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:41:36.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:41:36.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:41:36.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:41:36.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:41:36.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:41:36.780 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:41:37.216 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:41:37.216 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:41:37.216 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:41:37.216 INFO:tasks.workunit.client.0.smithi129.stderr:3' 2024-03-25T23:41:37.216 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:41:37.216 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:41:37.216 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:41:37.366 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836527 2024-03-25T23:41:37.366 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836527 2024-03-25T23:41:37.366 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527' 2024-03-25T23:41:37.366 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:41:37.366 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:41:37.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135878 2024-03-25T23:41:37.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135878 2024-03-25T23:41:37.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527 1-244813135878' 2024-03-25T23:41:37.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:41:37.580 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:41:37.723 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509482 2024-03-25T23:41:37.723 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509482 2024-03-25T23:41:37.723 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527 1-244813135878 2-64424509482' 2024-03-25T23:41:37.723 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:41:37.723 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:41:37.865 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=274877906946 2024-03-25T23:41:37.865 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 274877906946 2024-03-25T23:41:37.865 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527 1-244813135878 2-64424509482 3-274877906946' 2024-03-25T23:41:37.865 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:41:37.866 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836527 2024-03-25T23:41:37.866 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:41:37.866 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:41:37.867 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836527 2024-03-25T23:41:37.867 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:41:37.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836527 2024-03-25T23:41:37.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836527' 2024-03-25T23:41:37.869 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 21474836527 2024-03-25T23:41:37.869 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:41:38.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836527 -lt 21474836527 2024-03-25T23:41:38.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:41:38.172 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-244813135878 2024-03-25T23:41:38.172 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:41:38.174 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:41:38.174 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-244813135878 2024-03-25T23:41:38.175 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:41:38.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135878 2024-03-25T23:41:38.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 244813135878' 2024-03-25T23:41:38.176 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 244813135878 2024-03-25T23:41:38.176 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:41:38.544 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135879 -lt 244813135878 2024-03-25T23:41:38.545 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:41:38.545 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509482 2024-03-25T23:41:38.546 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:41:38.547 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:41:38.548 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509482 2024-03-25T23:41:38.548 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:41:38.549 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509482 2024-03-25T23:41:38.549 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509482' 2024-03-25T23:41:38.549 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 64424509482 2024-03-25T23:41:38.549 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:41:38.886 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509483 -lt 64424509482 2024-03-25T23:41:38.886 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:41:38.887 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-274877906946 2024-03-25T23:41:38.887 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:41:38.888 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:41:38.889 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-274877906946 2024-03-25T23:41:38.889 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:41:38.890 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=274877906946 2024-03-25T23:41:38.890 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 274877906946' 2024-03-25T23:41:38.891 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.3 seq 274877906946 2024-03-25T23:41:38.891 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:41:39.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 274877906946 2024-03-25T23:41:39.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:41:40.239 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:41:40.240 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:41:40.557 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 274877906946 -lt 274877906946 2024-03-25T23:41:40.557 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:41:40.558 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:41:40.558 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:41:40.935 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:41:40.935 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:41:40.936 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:41:40.936 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:41:40.936 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:41:40.936 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:41:40.937 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:41:40.937 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:41:41.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:41:41.261 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:41:41.262 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:41:41.262 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:41:41.611 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:41:41.611 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:41:41.611 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:41:41.611 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-25T23:41:41.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-25T23:41:41.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2024-03-25T23:41:41.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-03-25T23:41:41.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-03-25T23:41:41.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=1 2024-03-25T23:41:41.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=2 2024-03-25T23:41:41.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-03-25T23:41:41.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-25T23:41:41.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:41:41.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:41:41.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 182154"' 2024-03-25T23:41:41.664 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 182154' 2024-03-25T23:41:41.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-03-25T23:41:41.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:41:41.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:41:41.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-25T23:41:41.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 182156"' 2024-03-25T23:41:41.665 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 182156' 2024-03-25T23:41:41.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-03-25T23:41:41.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/182157: /' 2024-03-25T23:41:41.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 182154 182156' 2024-03-25T23:41:41.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-25T23:41:41.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-25T23:41:41.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 182154 2024-03-25T23:41:41.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-03-25T23:41:41.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/182159: /' 2024-03-25T23:41:44.807 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:41:44.807 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:41:44.808 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-03-25T23:41:44.808 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:41:44.808 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-03-25T23:41:44.808 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:41:44.808 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:41:44.808 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-03-25T23:41:44.808 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:41:44.808 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-25T23:41:44.808 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:41:44.808 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:41:44.808 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:41:44.808 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:41:44.808 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:41:44.809 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:41:44.809 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-03-25T23:41:44.809 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:41:44.809 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:41:44.809 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-03-25T23:41:44.809 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:41:44.809 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-03-25T23:41:44.809 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-03-25T23:41:44.809 INFO:tasks.workunit.client.0.smithi129.stderr:182159: remove 2#2:eb822e21:::SOMETHING:head# 2024-03-25T23:41:44.809 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-03-25T23:41:44.809 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:41:44.809 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:41:44.809 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-25T23:41:44.810 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:41:44.810 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-25T23:41:44.810 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:41:44.810 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:41:44.812 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:41:44.812 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:41:44.812 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-25T23:41:44.813 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-25T23:41:44.813 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:41:44.813 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:41:44.813 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:41:44.813 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:41:44.813 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:41:44.813 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:41:44.813 INFO:tasks.workunit.client.0.smithi129.stderr:182159: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:41:44.813 INFO:tasks.workunit.client.0.smithi129.stderr:182159: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:41:44.813 INFO:tasks.workunit.client.0.smithi129.stderr:182159: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:41:44.813 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:41:44.813 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:41:44.813 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:41:44.814 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:41:44.814 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:41:44.814 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:41:44.814 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:41:44.814 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:41:44.814 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:41:44.814 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:41:44.814 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:41:44.814 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-25T23:41:44.814 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-25T23:41:44.814 INFO:tasks.workunit.client.0.smithi129.stderr:182159: start osd.2 2024-03-25T23:41:44.814 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:41:45.339 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:41:45.340 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:41:45.340 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-25T23:41:45.340 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:41:45.340 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-25T23:41:45.340 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:41:45.340 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:41:45.340 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-25T23:41:45.340 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:41:45.340 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-25T23:41:45.340 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:41:45.340 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:41:45.340 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:41:45.341 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:41:45.341 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:41:45.341 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:41:45.341 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-25T23:41:45.341 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:41:45.341 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:41:45.341 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-25T23:41:45.341 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:41:45.341 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:41:45.341 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-25T23:41:45.341 INFO:tasks.workunit.client.0.smithi129.stderr:182157: remove 1#2:eb822e21:::SOMETHING:head# 2024-03-25T23:41:45.341 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-25T23:41:45.341 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:41:45.341 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:41:45.342 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-25T23:41:45.342 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:41:45.342 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:41:45.342 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:41:45.342 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:41:45.343 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:41:45.343 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:41:45.343 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-25T23:41:45.343 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-25T23:41:45.344 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:41:45.344 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:41:45.344 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:41:45.344 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:41:45.344 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:41:45.344 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:41:45.344 INFO:tasks.workunit.client.0.smithi129.stderr:182157: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:41:45.344 INFO:tasks.workunit.client.0.smithi129.stderr:182157: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:41:45.344 INFO:tasks.workunit.client.0.smithi129.stderr:182157: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:41:45.344 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:41:45.344 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:41:45.344 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:41:45.344 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:41:45.345 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:41:45.345 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:41:45.345 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:41:45.345 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:41:45.345 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:41:45.345 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:41:45.345 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:41:45.345 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-25T23:41:45.345 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-25T23:41:45.345 INFO:tasks.workunit.client.0.smithi129.stderr:182157: start osd.1 2024-03-25T23:41:45.345 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:41:49.429 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-25T23:41:49.429 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-25T23:41:49.429 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:41:49.429 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:41:49.429 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:41:49.429 INFO:tasks.workunit.client.0.smithi129.stderr:182159: 2024-03-25T23:41:44.844+0000 7fd23a77d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:41:49.430 INFO:tasks.workunit.client.0.smithi129.stderr:182159: 2024-03-25T23:41:44.848+0000 7fd23a77d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:41:49.430 INFO:tasks.workunit.client.0.smithi129.stderr:182159: 2024-03-25T23:41:44.848+0000 7fd23a77d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:41:49.430 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-25T23:41:49.430 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:41:49.430 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-25T23:41:49.430 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:41:49.430 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:41:49.430 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:41:49.430 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:41:49.430 INFO:tasks.workunit.client.0.smithi129.stderr:182159: 0 2024-03-25T23:41:49.430 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:41:49.430 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:41:49.430 INFO:tasks.workunit.client.0.smithi129.stderr:182159: 2024-03-25T23:41:45.436+0000 7fd23a77d7c0 -1 Falling back to public interface 2024-03-25T23:41:49.431 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:41:49.431 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:41:49.431 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:41:49.431 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:41:49.431 INFO:tasks.workunit.client.0.smithi129.stderr:182159: 1 2024-03-25T23:41:49.431 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:41:49.431 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:41:49.431 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:41:49.431 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:41:49.431 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:41:49.431 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:41:49.431 INFO:tasks.workunit.client.0.smithi129.stderr:182159: 2 2024-03-25T23:41:49.431 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:41:49.431 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:41:49.432 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:41:49.432 INFO:tasks.workunit.client.0.smithi129.stderr:182159: 2024-03-25T23:41:48.799+0000 7fd23a77d7c0 -1 osd.2 66 log_to_monitors true 2024-03-25T23:41:49.432 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:41:49.432 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:41:49.432 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:41:49.432 INFO:tasks.workunit.client.0.smithi129.stderr:182159: 3 2024-03-25T23:41:49.432 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:41:49.432 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:41:50.097 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helper182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-25T23:41:50.097 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-25T23:41:50.097 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:41:50.098 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:41:50.098 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:41:50.098 INFO:tasks.workunit.client.0.smithi129.stderr:182157: 2024-03-25T23:41:45.376+0000 7f00935907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:41:50.098 INFO:tasks.workunit.client.0.smithi129.stderr:182157: 2024-03-25T23:41:45.376+0000 7f00935907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:41:50.098 INFO:tasks.workunit.client.0.smithi129.stderr:182157: 2024-03-25T23:41:45.380+0000 7f00935907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:41:50.098 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-25T23:41:50.098 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:41:50.098 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T23:41:50.098 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:41:50.098 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:41:50.098 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:41:50.098 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:41:50.099 INFO:tasks.workunit.client.0.smithi129.stderr:182157: 0 2024-03-25T23:41:50.099 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:41:50.099 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:41:50.099 INFO:tasks.workunit.client.0.smithi129.stderr:182157: 2024-03-25T23:41:45.911+0000 7f00935907c0 -1 Falling back to public interface 2024-03-25T23:41:50.099 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:41:50.099 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:41:50.099 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:41:50.099 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:41:50.099 INFO:tasks.workunit.client.0.smithi129.stderr:182157: 1 2024-03-25T23:41:50.099 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:41:50.099 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:41:50.099 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:41:50.099 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:41:50.099 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:41:50.100 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:41:50.100 INFO:tasks.workunit.client.0.smithi129.stderr:182157: 2 2024-03-25T23:41:50.100 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:41:50.100 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:41:50.100 INFO:tasks.workunit.client.0.smithi129.stderr:182157: 2024-03-25T23:41:48.499+0000 7f00935907c0 -1 osd.1 66 log_to_monitors true 2024-03-25T23:41:50.100 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:41:50.100 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:41:50.100 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:41:50.100 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:41:50.100 INFO:tasks.workunit.client.0.smithi129.stderr:182157: 3 2024-03-25T23:41:50.100 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:41:50.100 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:41:51.113 INFO:tasks.workunit.client.0.smithi129.stderr:182157: osd.1 up in weight 1 up_from 72 up_thru 60 down_at 67 last_clean_interval [57,66) [v2:127.0.0.1:6818/823623754,v1:127.0.0.1:6819/823623754] [v2:127.0.0.1:6820/823623754,v1:127.0.0.1:6821/823623754] exists,up e1891468-d0f5-4d04-9e87-8f7a4ef38ae8 2024-03-25T23:41:51.113 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:41:51.113 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:41:51.113 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:41:51.114 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:41:51.114 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:41:51.114 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:41:51.114 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:41:51.114 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:41:51.114 INFO:tasks.workunit.client.0.smithi129.stderr:182157: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:41:51.114 INFO:tasks.workunit.client.0.smithi129.stderr:182157: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:41:51.114 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:41:51.114 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:41:51.114 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:41:51.114 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:41:51.114 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:41:51.114 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:41:51.115 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:41:51.115 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:41:51.115 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:41:51.115 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:41:51.115 INFO:tasks.workunit.client.0.smithi129.stderr:182157: 1 2024-03-25T23:41:51.115 INFO:tasks.workunit.client.0.smithi129.stderr:182157: 2 2024-03-25T23:41:51.115 INFO:tasks.workunit.client.0.smithi129.stderr:182157: 3' 2024-03-25T23:41:51.115 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:41:51.115 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:41:51.115 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:41:51.115 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836531 2024-03-25T23:41:51.115 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836531 2024-03-25T23:41:51.115 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531' 2024-03-25T23:41:51.115 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:41:51.116 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:41:51.116 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645314 2024-03-25T23:41:51.116 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645314 2024-03-25T23:41:51.116 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531 1-309237645314' 2024-03-25T23:41:51.116 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:41:51.524 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tells.sh:985: wait_for_osd: sleep 1 2024-03-25T23:41:51.525 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:41:51.525 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:41:51.525 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:41:51.525 INFO:tasks.workunit.client.0.smithi129.stderr:182159: 4 2024-03-25T23:41:51.525 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:41:51.525 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:41:51.525 INFO:tasks.workunit.client.0.smithi129.stderr:182159: osd.2 up in weight 1 up_from 73 up_thru 53 down_at 67 last_clean_interval [15,66) [v2:127.0.0.1:6810/1279746517,v1:127.0.0.1:6811/1279746517] [v2:127.0.0.1:6812/1279746517,v1:127.0.0.1:6813/1279746517] exists,up 51e83ed2-2c31-4ea0-b586-33c43ca2595f 2024-03-25T23:41:51.525 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:41:51.525 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:41:51.525 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:41:51.525 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:41:51.525 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:41:51.525 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:41:51.526 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:41:51.526 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:41:51.526 INFO:tasks.workunit.client.0.smithi129.stderr:182159: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:41:51.526 INFO:tasks.workunit.client.0.smithi129.stderr:182159: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:41:51.526 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:41:51.526 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:41:51.526 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:41:51.526 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:41:51.526 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:41:51.526 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:41:51.526 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:41:51.526 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:41:51.526 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:41:51.526 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:41:51.527 INFO:tasks.workunit.client.0.smithi129.stderr:182159: 1 2024-03-25T23:41:51.527 INFO:tasks.workunit.client.0.smithi129.stderr:182159: 2 2024-03-25T23:41:51.527 INFO:tasks.workunit.client.0.smithi129.stderr:182159: 3' 2024-03-25T23:41:51.527 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:41:51.527 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:41:51.527 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:41:51.527 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836532 2024-03-25T23:41:51.527 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836532 2024-03-25T23:41:51.527 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836532' 2024-03-25T23:41:51.527 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:41:52.813 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:41:52.814 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645315 2024-03-25T23:41:52.814 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645315 2024-03-25T23:41:52.814 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836532 1-309237645315' 2024-03-25T23:41:52.814 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:41:52.814 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:41:52.814 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612611 2024-03-25T23:41:52.814 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612611 2024-03-25T23:41:52.814 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836532 1-309237645315 2-313532612611' 2024-03-25T23:41:52.814 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:41:52.814 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:41:52.814 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=274877906951 2024-03-25T23:41:52.814 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 274877906951 2024-03-25T23:41:52.814 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836532 1-309237645315 2-313532612611 3-274877906951' 2024-03-25T23:41:52.815 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:41:52.815 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836532 2024-03-25T23:41:52.815 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:41:52.815 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:41:52.815 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836532 2024-03-25T23:41:52.815 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:41:52.815 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836532 2024-03-25T23:41:52.815 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836532' 2024-03-25T23:41:52.815 INFO:tasks.workunit.client.0.smithi129.stderr:182159: waiting osd.0 seq 21474836532 2024-03-25T23:41:52.815 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:41:52.815 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836532 -lt 21474836532 2024-03-25T23:41:52.815 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:41:52.815 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:41:52.816 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-309237645315 2024-03-25T23:41:52.816 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:41:52.816 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-309237645315 2024-03-25T23:41:52.816 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:41:52.816 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645315 2024-03-25T23:41:52.816 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 309237645315' 2024-03-25T23:41:52.816 INFO:tasks.workunit.client.0.smithi129.stderr:182159: waiting osd.1 seq 309237645315 2024-03-25T23:41:52.816 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:41:53.681 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:221 osd.2 flush_pg_stats 2024-03-25T23:41:53.681 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612610 2024-03-25T23:41:53.681 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612610 2024-03-25T23:41:53.681 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531 1-309237645314 2-313532612610' 2024-03-25T23:41:53.681 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:41:53.681 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:41:53.682 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=274877906950 2024-03-25T23:41:53.682 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 274877906950 2024-03-25T23:41:53.682 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531 1-309237645314 2-313532612610 3-274877906950' 2024-03-25T23:41:53.682 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:41:53.682 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836531 2024-03-25T23:41:53.682 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:41:53.682 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:41:53.682 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836531 2024-03-25T23:41:53.682 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:41:53.682 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836531 2024-03-25T23:41:53.682 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836531' 2024-03-25T23:41:53.682 INFO:tasks.workunit.client.0.smithi129.stderr:182157: waiting osd.0 seq 21474836531 2024-03-25T23:41:53.682 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:41:53.682 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836530 -lt 21474836531 2024-03-25T23:41:53.683 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:41:53.683 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:41:53.683 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:41:53.683 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836532 -lt 21474836531 2024-03-25T23:41:53.683 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:41:53.683 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-309237645314 2024-03-25T23:41:53.683 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:41:53.683 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:41:53.683 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-309237645314 2024-03-25T23:41:53.683 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:41:53.683 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645314 2024-03-25T23:41:53.683 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 309237645314' 2024-03-25T23:41:53.683 INFO:tasks.workunit.client.0.smithi129.stderr:182157: waiting osd.1 seq 309237645314 2024-03-25T23:41:53.684 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:41:53.684 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645315 -lt 309237645314 2024-03-25T23:41:53.684 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:41:55.081 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-313532612610 2024-03-25T23:41:55.081 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:41:55.081 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:41:55.081 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-313532612610 2024-03-25T23:41:55.081 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:41:55.081 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612610 2024-03-25T23:41:55.081 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 313532612610' 2024-03-25T23:41:55.081 INFO:tasks.workunit.client.0.smithi129.stderr:182157: waiting osd.2 seq 313532612610 2024-03-25T23:41:55.081 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:41:55.081 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612611 -lt 313532612610 2024-03-25T23:41:55.082 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:41:55.082 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-274877906950 2024-03-25T23:41:55.082 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:41:55.082 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:41:55.082 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-274877906950 2024-03-25T23:41:55.082 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:41:55.082 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=274877906950 2024-03-25T23:41:55.082 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 274877906950' 2024-03-25T23:41:55.082 INFO:tasks.workunit.client.0.smithi129.stderr:182157: waiting osd.3 seq 274877906950 2024-03-25T23:41:55.082 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:41:55.082 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 274877906951 -lt 274877906950 2024-03-25T23:41:55.082 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:41:55.082 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:41:55.082 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:41:55.083 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:41:55.083 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:41:55.083 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:41:55.083 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:41:55.083 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:41:55.083 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:41:55.083 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:41:55.083 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:41:55.083 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:41:55.083 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:41:55.132 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.clien4: flush_pg_stats: test 309237645315 -lt 309237645315 2024-03-25T23:41:55.132 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:41:55.132 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:41:55.132 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-313532612611 2024-03-25T23:41:55.132 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:41:55.132 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-313532612611 2024-03-25T23:41:55.132 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:41:55.132 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612611 2024-03-25T23:41:55.132 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 313532612611' 2024-03-25T23:41:55.132 INFO:tasks.workunit.client.0.smithi129.stderr:182159: waiting osd.2 seq 313532612611 2024-03-25T23:41:55.132 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:41:55.132 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612610 -lt 313532612611 2024-03-25T23:41:55.133 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:41:55.133 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:41:55.133 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:41:55.133 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612611 -lt 313532612611 2024-03-25T23:41:55.133 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:41:55.133 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-274877906951 2024-03-25T23:41:55.133 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:41:55.133 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:41:55.133 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-274877906951 2024-03-25T23:41:55.133 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:41:55.133 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=274877906951 2024-03-25T23:41:55.133 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 274877906951' 2024-03-25T23:41:55.133 INFO:tasks.workunit.client.0.smithi129.stderr:182159: waiting osd.3 seq 274877906951 2024-03-25T23:41:55.133 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:41:55.133 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 274877906951 -lt 274877906951 2024-03-25T23:41:55.134 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:41:55.134 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:41:55.134 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:41:55.134 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:41:55.134 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:41:55.134 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:41:55.134 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:41:55.134 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:41:55.480 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(containst.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:41:55.480 INFO:tasks.workunit.client.0.smithi129.stderr:182157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:41:55.480 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:41:55.480 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:41:55.481 INFO:tasks.workunit.client.0.smithi129.stderr:182157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:41:55.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:41:55.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-25T23:41:55.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 182156 2024-03-25T23:41:55.813 INFO:tasks.workunit.client.0.smithi129.stderr:("stale") | not)' 2024-03-25T23:41:55.813 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:41:55.813 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:41:55.813 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:41:55.813 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:41:55.813 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:41:55.813 INFO:tasks.workunit.client.0.smithi129.stderr:182159: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:41:55.813 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:41:55.813 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:41:55.813 INFO:tasks.workunit.client.0.smithi129.stderr:182159: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:41:55.813 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:41:55.813 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-25T23:41:55.814 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-25T23:41:55.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-25T23:41:55.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-03-25T23:41:55.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-25T23:41:55.814 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-03-25T23:41:55.815 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-25T23:41:55.815 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-25T23:41:55.815 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-25T23:41:55.815 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-25T23:41:56.184 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-03-25T23:41:56.184 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-03-25T23:41:56.184 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-25T23:41:56.185 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-25T23:41:56.185 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:41:56.185 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:41:56.185 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:41:56.185 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:41:56.525 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-25T23:41:25.991748+0000 2024-03-25T23:41:56.525 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-25T23:41:56.814 INFO:tasks.workunit.client.0.smithi129.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-25T23:41:56.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-25T23:41:25.991748+0000 2024-03-25T23:41:56.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-25T23:41:56.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-25T23:41:25.991748+0000 2024-03-25T23:41:56.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-25T23:41:56.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-25T23:41:56.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:41:56.831 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:41:56.831 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:41:56.831 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:41:56.832 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:41:56.832 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:41:57.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:41:25.991748+0000 '>' 2024-03-25T23:41:25.991748+0000 2024-03-25T23:41:57.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:41:58.141 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:41:58.141 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:41:58.142 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:41:58.142 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:41:58.142 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:41:58.142 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:41:58.142 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:41:58.455 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:41:25.991748+0000 '>' 2024-03-25T23:41:25.991748+0000 2024-03-25T23:41:58.455 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:41:59.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:41:59.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:41:59.457 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:41:59.457 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:41:59.457 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:41:59.458 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:41:59.458 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:41:59.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:41:25.991748+0000 '>' 2024-03-25T23:41:25.991748+0000 2024-03-25T23:41:59.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:42:00.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:42:00.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:42:00.782 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:42:00.782 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:42:00.782 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:42:00.783 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:42:00.783 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:42:01.100 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:41:25.991748+0000 '>' 2024-03-25T23:41:25.991748+0000 2024-03-25T23:42:01.101 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:42:02.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:42:02.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:42:02.103 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:42:02.103 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:42:02.103 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:42:02.103 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:42:02.104 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:42:02.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:41:57.395859+0000 '>' 2024-03-25T23:41:25.991748+0000 2024-03-25T23:42:02.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-25T23:42:02.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-03-25T23:42:02.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-25T23:42:02.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:42:02.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:42:02.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 185031"' 2024-03-25T23:42:02.419 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 185031' 2024-03-25T23:42:02.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-03-25T23:42:02.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:42:02.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:42:02.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 185033"' 2024-03-25T23:42:02.419 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 185033' 2024-03-25T23:42:02.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-03-25T23:42:02.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-25T23:42:02.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 185031 185033' 2024-03-25T23:42:02.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-25T23:42:02.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-25T23:42:02.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 185031 2024-03-25T23:42:02.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/185034: /' 2024-03-25T23:42:02.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-03-25T23:42:02.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/185036: /' 2024-03-25T23:42:03.392 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:42:03.392 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:42:03.392 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-25T23:42:03.392 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:42:03.392 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-25T23:42:03.392 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:42:03.392 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:42:03.392 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-25T23:42:03.392 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:42:03.392 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-25T23:42:03.393 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:42:03.393 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:42:03.393 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:42:03.393 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:42:03.393 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:42:03.393 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:42:03.393 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-25T23:42:03.393 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:42:03.393 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:42:03.393 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-25T23:42:03.393 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:42:03.393 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:42:03.393 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-25T23:42:03.393 INFO:tasks.workunit.client.0.smithi129.stderr:185034: _ 2024-03-25T23:42:03.394 INFO:tasks.workunit.client.0.smithi129.stderr:185034: hinfo_key 2024-03-25T23:42:03.394 INFO:tasks.workunit.client.0.smithi129.stderr:185034: snapset 2024-03-25T23:42:03.394 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-25T23:42:03.394 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:42:03.394 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:42:03.394 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-25T23:42:03.394 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:42:03.394 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:42:03.394 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:42:03.394 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:42:03.396 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:42:03.396 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:42:03.396 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-25T23:42:03.396 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-25T23:42:03.396 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:42:03.396 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:42:03.397 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:42:03.397 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:42:03.397 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:42:03.397 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:42:03.397 INFO:tasks.workunit.client.0.smithi129.stderr:185034: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:42:03.397 INFO:tasks.workunit.client.0.smithi129.stderr:185034: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:42:03.397 INFO:tasks.workunit.client.0.smithi129.stderr:185034: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:42:03.397 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:42:03.397 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:42:03.397 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:42:03.397 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:42:03.397 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:42:03.397 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:42:03.398 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:42:03.398 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:42:03.398 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:42:03.398 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:42:03.398 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:42:03.398 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-25T23:42:03.398 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-25T23:42:03.398 INFO:tasks.workunit.client.0.smithi129.stderr:185034: start osd.1 2024-03-25T23:42:03.426 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:42:03.427 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:42:03.427 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-03-25T23:42:03.427 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:42:03.427 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2024-03-25T23:42:03.427 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:42:03.427 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:42:03.427 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-03-25T23:42:03.427 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:42:03.427 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-25T23:42:03.427 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:42:03.427 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:42:03.427 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:42:03.427 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:42:03.428 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:42:03.428 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:42:03.428 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2024-03-25T23:42:03.428 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:42:03.428 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:42:03.428 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-03-25T23:42:03.428 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:42:03.428 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-03-25T23:42:03.428 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2024-03-25T23:42:03.428 INFO:tasks.workunit.client.0.smithi129.stderr:185036: _ 2024-03-25T23:42:03.428 INFO:tasks.workunit.client.0.smithi129.stderr:185036: hinfo_key 2024-03-25T23:42:03.428 INFO:tasks.workunit.client.0.smithi129.stderr:185036: snapset 2024-03-25T23:42:03.428 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-03-25T23:42:03.428 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:42:03.428 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:42:03.429 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-25T23:42:03.429 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:42:03.429 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-25T23:42:03.429 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:42:03.429 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:42:03.430 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:42:03.430 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:42:03.430 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-25T23:42:03.430 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-25T23:42:03.430 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:42:03.431 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:42:03.431 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:42:03.431 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:42:03.431 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:42:03.431 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:42:03.431 INFO:tasks.workunit.client.0.smithi129.stderr:185036: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:42:03.431 INFO:tasks.workunit.client.0.smithi129.stderr:185036: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:42:03.431 INFO:tasks.workunit.client.0.smithi129.stderr:185036: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:42:03.431 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:42:03.431 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:42:03.431 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:42:03.431 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:42:03.431 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:42:03.431 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:42:03.432 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:42:03.432 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:42:03.432 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:42:03.432 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:42:03.432 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:42:03.432 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-25T23:42:03.432 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-25T23:42:03.432 INFO:tasks.workunit.client.0.smithi129.stderr:185036: start osd.2 2024-03-25T23:42:08.344 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:42:08.344 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-25T23:42:08.344 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-25T23:42:08.344 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:42:08.344 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:42:08.344 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:42:08.344 INFO:tasks.workunit.client.0.smithi129.stderr:185036: 2024-03-25T23:42:03.475+0000 7f1f633da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:42:08.344 INFO:tasks.workunit.client.0.smithi129.stderr:185036: 2024-03-25T23:42:03.475+0000 7f1f633da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:42:08.345 INFO:tasks.workunit.client.0.smithi129.stderr:185036: 2024-03-25T23:42:03.479+0000 7f1f633da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:42:08.345 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-25T23:42:08.345 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:42:08.345 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-25T23:42:08.345 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:42:08.345 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:42:08.345 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:42:08.345 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:42:08.345 INFO:tasks.workunit.client.0.smithi129.stderr:185036: 0 2024-03-25T23:42:08.345 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:42:08.346 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:42:08.346 INFO:tasks.workunit.client.0.smithi129.stderr:185036: 2024-03-25T23:42:04.043+0000 7f1f633da7c0 -1 Falling back to public interface 2024-03-25T23:42:08.346 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:42:08.346 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:42:08.346 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:42:08.346 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:42:08.346 INFO:tasks.workunit.client.0.smithi129.stderr:185036: 1 2024-03-25T23:42:08.346 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:42:08.346 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:42:08.346 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:42:08.347 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:42:08.347 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:42:08.347 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:42:08.347 INFO:tasks.workunit.client.0.smithi129.stderr:185036: 2 2024-03-25T23:42:08.347 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:42:08.347 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:42:08.347 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:42:08.347 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:42:08.347 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:42:08.347 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:42:08.347 INFO:tasks.workunit.client.0.smithi129.stderr:185036: 3 2024-03-25T23:42:08.347 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:42:08.347 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:42:08.347 INFO:tasks.workunit.client.0.smithi129.stderr:185036: 2024-03-25T23:42:07.875+0000 7f1f633da7c0 -1 osd.2 76 log_to_monitors true 2024-03-25T23:42:08.348 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/covery_ops 2024-03-25T23:42:08.348 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-25T23:42:08.348 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-25T23:42:08.348 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:42:08.348 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:42:08.348 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:42:08.348 INFO:tasks.workunit.client.0.smithi129.stderr:185034: 2024-03-25T23:42:03.427+0000 7f48cb69c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:42:08.348 INFO:tasks.workunit.client.0.smithi129.stderr:185034: 2024-03-25T23:42:03.447+0000 7f48cb69c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:42:08.348 INFO:tasks.workunit.client.0.smithi129.stderr:185034: 2024-03-25T23:42:03.451+0000 7f48cb69c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:42:08.348 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-25T23:42:08.348 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:42:08.348 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T23:42:08.348 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:42:08.349 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:42:08.349 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:42:08.349 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:42:08.349 INFO:tasks.workunit.client.0.smithi129.stderr:185034: 0 2024-03-25T23:42:08.349 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:42:08.349 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:42:08.349 INFO:tasks.workunit.client.0.smithi129.stderr:185034: 2024-03-25T23:42:04.015+0000 7f48cb69c7c0 -1 Falling back to public interface 2024-03-25T23:42:08.349 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:42:08.349 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:42:08.349 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:42:08.349 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:42:08.349 INFO:tasks.workunit.client.0.smithi129.stderr:185034: 1 2024-03-25T23:42:08.349 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:42:08.349 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:42:08.350 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:42:08.350 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:42:08.350 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:42:08.350 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:42:08.350 INFO:tasks.workunit.client.0.smithi129.stderr:185034: 2 2024-03-25T23:42:08.350 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:42:08.350 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:42:08.350 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:42:08.350 INFO:tasks.workunit.client.0.smithi129.stderr:185034: 2024-03-25T23:42:07.735+0000 7f48cb69c7c0 -1 osd.1 76 log_to_monitors true 2024-03-25T23:42:08.350 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:42:08.350 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:42:08.350 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:42:08.350 INFO:tasks.workunit.client.0.smithi129.stderr:185034: 3 2024-03-25T23:42:08.351 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:42:08.351 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:42:10.356 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:42:10.356 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:42:10.356 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:42:10.356 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:42:10.356 INFO:tasks.workunit.client.0.smithi129.stderr:185034: 4 2024-03-25T23:42:10.356 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:42:10.356 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:42:10.357 INFO:tasks.workunit.client.0.smithi129.stderr:185034: osd.1 up in weight 1 up_from 82 up_thru 82 down_at 77 last_clean_interval [72,76) [v2:127.0.0.1:6810/3008127404,v1:127.0.0.1:6811/3008127404] [v2:127.0.0.1:6812/3008127404,v1:127.0.0.1:6813/3008127404] exists,up e1891468-d0f5-4d04-9e87-8f7a4ef38ae8 2024-03-25T23:42:10.357 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:42:10.357 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:42:10.357 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:42:10.357 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:42:10.357 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:42:10.357 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:42:10.357 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:42:10.357 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:42:10.357 INFO:tasks.workunit.client.0.smithi129.stderr:185034: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:42:10.357 INFO:tasks.workunit.client.0.smithi129.stderr:185034: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:42:10.357 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:42:10.357 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:42:10.357 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:42:10.358 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:42:10.358 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:42:10.358 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:42:10.358 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:42:10.358 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:42:10.358 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:42:10.358 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:42:10.358 INFO:tasks.workunit.client.0.smithi129.stderr:185034: 1 2024-03-25T23:42:10.358 INFO:tasks.workunit.client.0.smithi129.stderr:185034: 2 2024-03-25T23:42:10.358 INFO:tasks.workunit.client.0.smithi129.stderr:185034: 3' 2024-03-25T23:42:10.358 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:42:10.358 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:42:10.358 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:42:10.359 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836537 2024-03-25T23:42:10.359 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836537 2024-03-25T23:42:10.359 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836537' 2024-03-25T23:42:10.359 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:42:10.422 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standaloneph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:42:10.422 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:42:10.422 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:42:10.422 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:42:10.422 INFO:tasks.workunit.client.0.smithi129.stderr:185036: 4 2024-03-25T23:42:10.422 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:42:10.422 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:42:10.422 INFO:tasks.workunit.client.0.smithi129.stderr:185036: osd.2 up in weight 1 up_from 82 up_thru 53 down_at 77 last_clean_interval [73,76) [v2:127.0.0.1:6818/2375575082,v1:127.0.0.1:6819/2375575082] [v2:127.0.0.1:6820/2375575082,v1:127.0.0.1:6821/2375575082] exists,up 51e83ed2-2c31-4ea0-b586-33c43ca2595f 2024-03-25T23:42:10.422 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:42:10.422 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:42:10.422 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:42:10.422 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:42:10.423 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:42:10.423 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:42:10.423 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:42:10.423 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:42:10.423 INFO:tasks.workunit.client.0.smithi129.stderr:185036: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:42:10.423 INFO:tasks.workunit.client.0.smithi129.stderr:185036: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:42:10.423 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:42:10.423 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:42:10.423 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:42:10.423 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:42:10.423 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:42:10.423 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:42:10.423 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:42:10.424 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:42:10.424 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:42:10.424 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:42:10.424 INFO:tasks.workunit.client.0.smithi129.stderr:185036: 1 2024-03-25T23:42:10.424 INFO:tasks.workunit.client.0.smithi129.stderr:185036: 2 2024-03-25T23:42:10.424 INFO:tasks.workunit.client.0.smithi129.stderr:185036: 3' 2024-03-25T23:42:10.424 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:42:10.424 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:42:10.424 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:42:10.424 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836538 2024-03-25T23:42:10.424 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836538 2024-03-25T23:42:10.424 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836538' 2024-03-25T23:42:10.424 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:42:12.371 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:42:12.371 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318274 2024-03-25T23:42:12.372 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318274 2024-03-25T23:42:12.372 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836537 1-352187318274' 2024-03-25T23:42:12.372 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:42:12.372 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:42:12.372 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318274 2024-03-25T23:42:12.372 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318274 2024-03-25T23:42:12.372 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836537 1-352187318274 2-352187318274' 2024-03-25T23:42:12.372 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:42:12.372 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:42:12.372 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=274877906955 2024-03-25T23:42:12.372 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 274877906955 2024-03-25T23:42:12.372 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836537 1-352187318274 2-352187318274 3-274877906955' 2024-03-25T23:42:12.372 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:42:12.372 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836537 2024-03-25T23:42:12.373 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:42:12.373 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:42:12.373 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836537 2024-03-25T23:42:12.373 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:42:12.373 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836537 2024-03-25T23:42:12.373 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836537' 2024-03-25T23:42:12.373 INFO:tasks.workunit.client.0.smithi129.stderr:185034: waiting osd.0 seq 21474836537 2024-03-25T23:42:12.373 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:42:12.373 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836536 -lt 21474836537 2024-03-25T23:42:12.373 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:42:12.373 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:42:12.373 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:42:12.373 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836538 -lt 21474836537 2024-03-25T23:42:12.373 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:42:12.373 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:42:12.374 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-352187318274 2024-03-25T23:42:12.374 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:42:12.374 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-352187318274 2024-03-25T23:42:12.374 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:42:12.456 INFO:tasks.workunit.client.0.smithi129.stderr:18503e/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:42:12.457 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318275 2024-03-25T23:42:12.457 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318275 2024-03-25T23:42:12.457 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836538 1-352187318275' 2024-03-25T23:42:12.457 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:42:12.457 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:42:12.457 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318275 2024-03-25T23:42:12.457 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318275 2024-03-25T23:42:12.457 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836538 1-352187318275 2-352187318275' 2024-03-25T23:42:12.457 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:42:12.457 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:42:12.457 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=274877906957 2024-03-25T23:42:12.457 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 274877906957 2024-03-25T23:42:12.457 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836538 1-352187318275 2-352187318275 3-274877906957' 2024-03-25T23:42:12.458 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:42:12.458 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836538 2024-03-25T23:42:12.458 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:42:12.458 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:42:12.458 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836538 2024-03-25T23:42:12.458 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:42:12.458 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836538 2024-03-25T23:42:12.458 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836538' 2024-03-25T23:42:12.458 INFO:tasks.workunit.client.0.smithi129.stderr:185036: waiting osd.0 seq 21474836538 2024-03-25T23:42:12.458 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:42:12.458 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836536 -lt 21474836538 2024-03-25T23:42:12.458 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:42:12.458 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:42:12.458 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:42:12.459 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836538 -lt 21474836538 2024-03-25T23:42:12.459 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:42:12.459 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-352187318275 2024-03-25T23:42:12.459 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:42:12.459 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:42:12.459 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-352187318275 2024-03-25T23:42:12.459 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:42:13.758 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318274 2024-03-25T23:42:13.758 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 352187318274' 2024-03-25T23:42:13.759 INFO:tasks.workunit.client.0.smithi129.stderr:185034: waiting osd.1 seq 352187318274 2024-03-25T23:42:13.759 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:42:13.759 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318275 -lt 352187318274 2024-03-25T23:42:13.759 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:42:13.759 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-352187318274 2024-03-25T23:42:13.759 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:42:13.759 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:42:13.759 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-352187318274 2024-03-25T23:42:13.759 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:42:13.759 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318274 2024-03-25T23:42:13.759 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 352187318274' 2024-03-25T23:42:13.759 INFO:tasks.workunit.client.0.smithi129.stderr:185034: waiting osd.2 seq 352187318274 2024-03-25T23:42:13.759 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:42:13.760 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318275 -lt 352187318274 2024-03-25T23:42:13.760 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:42:13.760 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:42:13.760 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-274877906955 2024-03-25T23:42:13.760 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:42:13.760 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-274877906955 2024-03-25T23:42:13.760 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:42:13.760 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=274877906955 2024-03-25T23:42:13.760 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 274877906955' 2024-03-25T23:42:13.760 INFO:tasks.workunit.client.0.smithi129.stderr:185034: waiting osd.3 seq 274877906955 2024-03-25T23:42:13.760 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:42:13.760 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 274877906957 -lt 274877906955 2024-03-25T23:42:13.760 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:42:13.760 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:42:13.761 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:42:13.761 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:42:13.761 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:42:13.761 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:42:13.761 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:42:13.761 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:42:13.922 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(6: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318275 2024-03-25T23:42:13.922 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 352187318275' 2024-03-25T23:42:13.922 INFO:tasks.workunit.client.0.smithi129.stderr:185036: waiting osd.1 seq 352187318275 2024-03-25T23:42:13.922 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:42:13.922 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318275 -lt 352187318275 2024-03-25T23:42:13.923 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:42:13.923 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-352187318275 2024-03-25T23:42:13.923 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:42:13.923 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:42:13.923 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-352187318275 2024-03-25T23:42:13.923 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:42:13.924 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318275 2024-03-25T23:42:13.924 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 352187318275' 2024-03-25T23:42:13.924 INFO:tasks.workunit.client.0.smithi129.stderr:185036: waiting osd.2 seq 352187318275 2024-03-25T23:42:13.924 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:42:13.924 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318275 -lt 352187318275 2024-03-25T23:42:13.924 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:42:13.924 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-274877906957 2024-03-25T23:42:13.924 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:42:13.924 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:42:13.924 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:42:13.925 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-274877906957 2024-03-25T23:42:13.925 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=274877906957 2024-03-25T23:42:13.925 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 274877906957' 2024-03-25T23:42:13.925 INFO:tasks.workunit.client.0.smithi129.stderr:185036: waiting osd.3 seq 274877906957 2024-03-25T23:42:13.925 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:42:13.925 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 274877906957 -lt 274877906957 2024-03-25T23:42:13.925 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:42:13.926 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:42:13.926 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:42:13.926 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:42:13.926 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:42:13.926 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:42:13.926 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:42:13.927 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:42:14.427 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:42:14.427 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:42:14.427 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:42:14.428 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:42:14.428 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:42:14.428 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:42:14.428 INFO:tasks.workunit.client.0.smithi129.stderr:185034: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:42:14.428 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:42:14.428 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:42:14.428 INFO:tasks.workunit.client.0.smithi129.stderr:185034: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:42:14.428 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:42:14.428 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-25T23:42:14.428 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 185033 2024-03-25T23:42:14.582 INFO:tasks.workunit.client.0.smithi129.stderr:contains("stale") | not)' 2024-03-25T23:42:14.582 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:42:14.582 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:42:14.582 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:42:14.582 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:42:14.582 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:42:14.583 INFO:tasks.workunit.client.0.smithi129.stderr:185036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:42:14.583 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:42:14.583 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:42:14.583 INFO:tasks.workunit.client.0.smithi129.stderr:185036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:42:14.583 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:42:14.583 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-25T23:42:14.583 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-25T23:42:14.584 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-25T23:42:14.584 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-03-25T23:42:14.584 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-25T23:42:14.584 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-25T23:42:14.628 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-25T23:42:14.632 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2024-03-25T23:42:14.633 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-03-25T23:42:14.633 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-03-25T23:42:14.633 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-03-25T23:42:14.633 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=1 2024-03-25T23:42:14.633 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-03-25T23:42:14.633 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-25T23:42:14.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:42:14.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:42:14.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 187394"' 2024-03-25T23:42:14.634 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 187394' 2024-03-25T23:42:14.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-25T23:42:14.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:42:14.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:42:14.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-25T23:42:14.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 187396"' 2024-03-25T23:42:14.634 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 187396' 2024-03-25T23:42:14.635 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/187397: /' 2024-03-25T23:42:14.635 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-03-25T23:42:14.635 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 187394 187396' 2024-03-25T23:42:14.635 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-25T23:42:14.635 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-25T23:42:14.635 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 187394 2024-03-25T23:42:14.635 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-25T23:42:14.635 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/187400: /' 2024-03-25T23:42:17.940 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:42:17.940 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:42:17.940 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-25T23:42:17.940 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:42:17.940 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-03-25T23:42:17.940 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:42:17.940 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:42:17.940 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-25T23:42:17.940 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:42:17.940 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-25T23:42:17.940 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:42:17.941 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:42:17.941 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:42:17.941 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:42:17.941 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:42:17.941 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:42:17.941 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-03-25T23:42:17.941 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:42:17.941 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:42:17.941 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-25T23:42:17.941 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:42:17.941 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-25T23:42:17.941 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-03-25T23:42:17.941 INFO:tasks.workunit.client.0.smithi129.stderr:187397: remove 0#2:eb822e21:::SOMETHING:head# 2024-03-25T23:42:17.941 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-25T23:42:17.942 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:42:17.942 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:42:17.942 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-25T23:42:17.942 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:42:17.942 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-25T23:42:17.942 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:42:17.942 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:42:17.945 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:42:17.945 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:42:17.945 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-25T23:42:17.946 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-25T23:42:17.946 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:42:17.947 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:42:17.947 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:42:17.947 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:42:17.947 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:42:17.947 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:42:17.947 INFO:tasks.workunit.client.0.smithi129.stderr:187397: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:42:17.947 INFO:tasks.workunit.client.0.smithi129.stderr:187397: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:42:17.947 INFO:tasks.workunit.client.0.smithi129.stderr:187397: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:42:17.947 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:42:17.948 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:42:17.948 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:42:17.948 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:42:17.948 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:42:17.948 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:42:17.948 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:42:17.948 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:42:17.948 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:42:17.948 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:42:17.948 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:42:17.948 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-25T23:42:17.948 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-25T23:42:17.948 INFO:tasks.workunit.client.0.smithi129.stderr:187397: start osd.3 2024-03-25T23:42:17.948 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:42:18.270 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:42:18.270 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:42:18.270 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-25T23:42:18.270 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:42:18.270 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-25T23:42:18.270 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:42:18.271 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:42:18.271 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-25T23:42:18.271 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:42:18.271 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-25T23:42:18.271 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:42:18.271 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:42:18.271 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:42:18.271 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:42:18.271 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:42:18.271 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:42:18.271 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-25T23:42:18.271 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:42:18.271 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:42:18.272 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-25T23:42:18.272 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:42:18.272 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:42:18.272 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-25T23:42:18.272 INFO:tasks.workunit.client.0.smithi129.stderr:187400: remove 1#2:eb822e21:::SOMETHING:head# 2024-03-25T23:42:18.272 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-25T23:42:18.272 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:42:18.272 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:42:18.272 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-25T23:42:18.272 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:42:18.272 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:42:18.272 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:42:18.272 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:42:18.273 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:42:18.273 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:42:18.273 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-25T23:42:18.273 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-25T23:42:18.273 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:42:18.273 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:42:18.274 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:42:18.274 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:42:18.274 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:42:18.274 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:42:18.274 INFO:tasks.workunit.client.0.smithi129.stderr:187400: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:42:18.274 INFO:tasks.workunit.client.0.smithi129.stderr:187400: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:42:18.274 INFO:tasks.workunit.client.0.smithi129.stderr:187400: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:42:18.274 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:42:18.274 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:42:18.274 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:42:18.274 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:42:18.274 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:42:18.274 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:42:18.274 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:42:18.275 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:42:18.275 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:42:18.275 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:42:18.275 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:42:18.275 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-25T23:42:18.275 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-25T23:42:18.275 INFO:tasks.workunit.client.0.smithi129.stderr:187400: start osd.1 2024-03-25T23:42:18.275 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:42:22.612 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-25T23:42:22.612 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-25T23:42:22.612 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:42:22.612 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:42:22.612 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:42:22.612 INFO:tasks.workunit.client.0.smithi129.stderr:187397: 2024-03-25T23:42:17.979+0000 7f3ba2f857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:42:22.612 INFO:tasks.workunit.client.0.smithi129.stderr:187397: 2024-03-25T23:42:17.979+0000 7f3ba2f857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:42:22.612 INFO:tasks.workunit.client.0.smithi129.stderr:187397: 2024-03-25T23:42:17.983+0000 7f3ba2f857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:42:22.612 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-25T23:42:22.612 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:42:22.613 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-25T23:42:22.613 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:42:22.613 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:42:22.613 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:42:22.613 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:42:22.613 INFO:tasks.workunit.client.0.smithi129.stderr:187397: 0 2024-03-25T23:42:22.613 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:42:22.613 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:42:22.613 INFO:tasks.workunit.client.0.smithi129.stderr:187397: 2024-03-25T23:42:18.527+0000 7f3ba2f857c0 -1 Falling back to public interface 2024-03-25T23:42:22.613 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:42:22.613 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:42:22.613 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:42:22.613 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:42:22.614 INFO:tasks.workunit.client.0.smithi129.stderr:187397: 1 2024-03-25T23:42:22.614 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:42:22.614 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:42:22.614 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:42:22.614 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:42:22.614 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:42:22.614 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:42:22.614 INFO:tasks.workunit.client.0.smithi129.stderr:187397: 2 2024-03-25T23:42:22.614 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:42:22.614 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:42:22.614 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:42:22.614 INFO:tasks.workunit.client.0.smithi129.stderr:187397: 2024-03-25T23:42:21.479+0000 7f3ba2f857c0 -1 osd.3 83 log_to_monitors true 2024-03-25T23:42:22.614 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:42:22.615 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:42:22.615 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:42:22.615 INFO:tasks.workunit.client.0.smithi129.stderr:187397: 3 2024-03-25T23:42:22.615 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:42:22.615 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:42:23.075 INFO:tasks.workunit.client.0.smithi129.stderr:187397: osd.3 up in weight 1 up_from 88 up_thru 82 down_at 84 last_187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-25T23:42:23.075 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-25T23:42:23.075 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:42:23.075 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:42:23.075 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:42:23.075 INFO:tasks.workunit.client.0.smithi129.stderr:187400: 2024-03-25T23:42:18.299+0000 7f77f90017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:42:23.075 INFO:tasks.workunit.client.0.smithi129.stderr:187400: 2024-03-25T23:42:18.327+0000 7f77f90017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:42:23.075 INFO:tasks.workunit.client.0.smithi129.stderr:187400: 2024-03-25T23:42:18.331+0000 7f77f90017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:42:23.075 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-25T23:42:23.075 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:42:23.076 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T23:42:23.076 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:42:23.076 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:42:23.076 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:42:23.076 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:42:23.076 INFO:tasks.workunit.client.0.smithi129.stderr:187400: 0 2024-03-25T23:42:23.076 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:42:23.076 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:42:23.076 INFO:tasks.workunit.client.0.smithi129.stderr:187400: 2024-03-25T23:42:18.863+0000 7f77f90017c0 -1 Falling back to public interface 2024-03-25T23:42:23.076 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:42:23.076 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:42:23.076 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:42:23.077 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:42:23.077 INFO:tasks.workunit.client.0.smithi129.stderr:187400: 1 2024-03-25T23:42:23.077 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:42:23.077 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:42:23.077 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:42:23.077 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:42:23.077 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:42:23.077 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:42:23.077 INFO:tasks.workunit.client.0.smithi129.stderr:187400: 2 2024-03-25T23:42:23.077 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:42:23.077 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:42:23.077 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:42:23.077 INFO:tasks.workunit.client.0.smithi129.stderr:187400: 2024-03-25T23:42:22.047+0000 7f77f90017c0 -1 osd.1 83 log_to_monitors true 2024-03-25T23:42:23.078 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:42:23.078 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:42:23.078 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:42:23.078 INFO:tasks.workunit.client.0.smithi129.stderr:187400: 3 2024-03-25T23:42:23.078 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:42:23.078 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:42:23.631 INFO:tasks.workunit.client.0.smithi129.stderr:187400: osd.1 up in weight 1 up_from 89 up_thru 82 down_at 84 last_clean_interval [64,83) [v2:127.0.0.1:6810/696700860,v1:127.0.0.1:6811/696700860] [v2:127.0.0.1:6812/696700860,v1:127.0.0.1:6813/696700860] exists,up 2de53dc4-8cd5-4295-9d23-e6a36c158ac1 2024-03-25T23:42:23.631 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:42:23.631 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:42:23.631 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:42:23.631 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:42:23.631 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:42:23.631 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:42:23.632 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:42:23.632 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:42:23.632 INFO:tasks.workunit.client.0.smithi129.stderr:187397: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:42:23.632 INFO:tasks.workunit.client.0.smithi129.stderr:187397: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:42:23.632 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:42:23.632 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:42:23.632 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:42:23.632 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:42:23.632 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:42:23.632 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:42:23.632 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:42:23.632 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:42:23.632 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:42:23.632 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:42:23.633 INFO:tasks.workunit.client.0.smithi129.stderr:187397: 1 2024-03-25T23:42:23.633 INFO:tasks.workunit.client.0.smithi129.stderr:187397: 2 2024-03-25T23:42:23.633 INFO:tasks.workunit.client.0.smithi129.stderr:187397: 3' 2024-03-25T23:42:23.633 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:42:23.633 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:42:23.633 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:42:23.633 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836542 2024-03-25T23:42:23.633 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836542 2024-03-25T23:42:23.633 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836542' 2024-03-25T23:42:23.633 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:42:23.633 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:42:23.633 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=382252089346 2024-03-25T23:42:23.633 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 382252089346 2024-03-25T23:42:23.633 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836542 1-382252089346' 2024-03-25T23:42:23.633 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:42:24.067 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tellclean_interval [82,83) [v2:127.0.0.1:6826/1998584098,v1:127.0.0.1:6827/1998584098] [v2:127.0.0.1:6828/1998584098,v1:127.0.0.1:6829/1998584098] exists,up e1891468-d0f5-4d04-9e87-8f7a4ef38ae8 2024-03-25T23:42:24.068 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:42:24.068 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:42:24.068 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:42:24.068 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:42:24.068 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:42:24.068 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:42:24.068 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:42:24.068 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:42:24.068 INFO:tasks.workunit.client.0.smithi129.stderr:187400: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:42:24.068 INFO:tasks.workunit.client.0.smithi129.stderr:187400: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:42:24.068 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:42:24.069 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:42:24.069 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:42:24.069 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:42:24.069 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:42:24.069 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:42:24.069 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:42:24.069 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:42:24.069 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:42:24.069 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:42:24.069 INFO:tasks.workunit.client.0.smithi129.stderr:187400: 1 2024-03-25T23:42:24.069 INFO:tasks.workunit.client.0.smithi129.stderr:187400: 2 2024-03-25T23:42:24.069 INFO:tasks.workunit.client.0.smithi129.stderr:187400: 3' 2024-03-25T23:42:24.069 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:42:24.070 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:42:24.070 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:42:24.070 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836543 2024-03-25T23:42:24.070 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836543 2024-03-25T23:42:24.070 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543' 2024-03-25T23:42:24.070 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:42:24.070 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:42:24.071 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=382252089347 2024-03-25T23:42:24.071 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 382252089347 2024-03-25T23:42:24.071 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543 1-382252089347' 2024-03-25T23:42:24.071 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:42:25.054 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph osd.2 flush_pg_stats 2024-03-25T23:42:25.055 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318279 2024-03-25T23:42:25.055 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318279 2024-03-25T23:42:25.055 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836542 1-382252089346 2-352187318279' 2024-03-25T23:42:25.055 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:42:25.055 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:42:25.055 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122050 2024-03-25T23:42:25.055 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122050 2024-03-25T23:42:25.055 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836542 1-382252089346 2-352187318279 3-377957122050' 2024-03-25T23:42:25.055 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:42:25.055 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836542 2024-03-25T23:42:25.055 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:42:25.056 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:42:25.056 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836542 2024-03-25T23:42:25.056 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:42:25.056 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836542 2024-03-25T23:42:25.056 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836542' 2024-03-25T23:42:25.056 INFO:tasks.workunit.client.0.smithi129.stderr:187397: waiting osd.0 seq 21474836542 2024-03-25T23:42:25.056 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:42:25.056 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836542 -lt 21474836542 2024-03-25T23:42:25.056 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:42:25.056 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-382252089346 2024-03-25T23:42:25.056 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:42:25.056 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:42:25.056 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-382252089346 2024-03-25T23:42:25.056 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:42:25.057 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=382252089346 2024-03-25T23:42:25.057 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 382252089346' 2024-03-25T23:42:25.057 INFO:tasks.workunit.client.0.smithi129.stderr:187397: waiting osd.1 seq 382252089346 2024-03-25T23:42:25.057 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:42:25.057 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 382252089346 -lt 382252089346 2024-03-25T23:42:25.057 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:42:25.057 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-352187318279 2024-03-25T23:42:25.057 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:42:25.057 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:42:25.057 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-352187318279 2024-03-25T23:42:26.719 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtesttell osd.2 flush_pg_stats 2024-03-25T23:42:26.720 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318280 2024-03-25T23:42:26.720 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318280 2024-03-25T23:42:26.720 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543 1-382252089347 2-352187318280' 2024-03-25T23:42:26.720 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:42:26.720 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:42:26.720 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122051 2024-03-25T23:42:26.720 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122051 2024-03-25T23:42:26.720 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543 1-382252089347 2-352187318280 3-377957122051' 2024-03-25T23:42:26.720 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:42:26.720 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836543 2024-03-25T23:42:26.720 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:42:26.720 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:42:26.720 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836543 2024-03-25T23:42:26.721 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:42:26.721 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836543 2024-03-25T23:42:26.721 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836543' 2024-03-25T23:42:26.721 INFO:tasks.workunit.client.0.smithi129.stderr:187400: waiting osd.0 seq 21474836543 2024-03-25T23:42:26.721 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:42:26.721 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836542 -lt 21474836543 2024-03-25T23:42:26.721 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:42:26.721 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:42:26.721 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:42:26.721 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836543 -lt 21474836543 2024-03-25T23:42:26.721 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:42:26.721 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-382252089347 2024-03-25T23:42:26.722 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:42:26.722 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:42:26.722 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-382252089347 2024-03-25T23:42:26.722 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:42:26.722 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=382252089347 2024-03-25T23:42:26.722 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 382252089347' 2024-03-25T23:42:26.722 INFO:tasks.workunit.client.0.smithi129.stderr:187400: waiting osd.1 seq 382252089347 2024-03-25T23:42:26.722 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:42:26.722 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 382252089347 -lt 382252089347 2024-03-25T23:42:26.722 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:42:27.778 INFO:tasks.workunit.client.0.smithi129.stderr:1874/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:42:27.778 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318279 2024-03-25T23:42:27.778 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 352187318279' 2024-03-25T23:42:27.778 INFO:tasks.workunit.client.0.smithi129.stderr:187397: waiting osd.2 seq 352187318279 2024-03-25T23:42:27.778 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:42:27.778 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318278 -lt 352187318279 2024-03-25T23:42:27.778 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:42:27.779 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:42:27.779 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:42:27.779 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318280 -lt 352187318279 2024-03-25T23:42:27.779 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:42:27.779 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-377957122050 2024-03-25T23:42:27.779 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:42:27.779 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:42:27.779 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-377957122050 2024-03-25T23:42:27.779 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:42:27.779 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122050 2024-03-25T23:42:27.779 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 377957122050' 2024-03-25T23:42:27.779 INFO:tasks.workunit.client.0.smithi129.stderr:187397: waiting osd.3 seq 377957122050 2024-03-25T23:42:27.779 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:42:27.779 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122051 -lt 377957122050 2024-03-25T23:42:27.779 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:42:27.780 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:42:27.780 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:42:27.780 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:42:27.780 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:42:27.780 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:42:27.780 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:42:27.780 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:42:27.780 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:42:27.780 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:42:27.780 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:42:27.780 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-25T23:42:27.780 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:42:28.144 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.cli00: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-352187318280 2024-03-25T23:42:28.144 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:42:28.144 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:42:28.145 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-352187318280 2024-03-25T23:42:28.145 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:42:28.145 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318280 2024-03-25T23:42:28.145 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 352187318280' 2024-03-25T23:42:28.145 INFO:tasks.workunit.client.0.smithi129.stderr:187400: waiting osd.2 seq 352187318280 2024-03-25T23:42:28.145 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:42:28.145 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318280 -lt 352187318280 2024-03-25T23:42:28.145 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:42:28.145 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-377957122051 2024-03-25T23:42:28.145 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:42:28.145 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:42:28.145 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-377957122051 2024-03-25T23:42:28.145 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:42:28.145 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122051 2024-03-25T23:42:28.146 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 377957122051' 2024-03-25T23:42:28.146 INFO:tasks.workunit.client.0.smithi129.stderr:187400: waiting osd.3 seq 377957122051 2024-03-25T23:42:28.146 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:42:28.146 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122051 -lt 377957122051 2024-03-25T23:42:28.146 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:42:28.146 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:42:28.146 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:42:28.146 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:42:28.146 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:42:28.146 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:42:28.146 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:42:28.146 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:42:28.146 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:42:28.146 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:42:28.147 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:42:28.147 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T23:42:28.147 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:42:29.082 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.cent.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:42:29.082 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:42:29.083 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 5 2024-03-25T23:42:29.083 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' -1 2024-03-25T23:42:29.083 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-25T23:42:29.083 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=1 2024-03-25T23:42:29.083 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:42:29.083 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-25T23:42:29.083 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:42:29.083 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:42:29.083 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:42:29.083 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:42:29.083 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:42:29.083 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:42:29.083 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:42:29.083 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:42:29.084 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T23:42:29.084 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:42:29.084 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:42:29.084 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:42:29.084 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-25T23:42:29.084 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 1 2024-03-25T23:42:29.084 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-25T23:42:29.084 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-25T23:42:29.084 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:42:29.084 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-25T23:42:29.084 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:42:29.084 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:42:29.084 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:42:29.084 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:42:29.084 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:42:29.085 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:42:29.085 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:42:29.743 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:13lient.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:42:29.744 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:42:29.744 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-25T23:42:29.744 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-25T23:42:29.744 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-25T23:42:29.744 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-25T23:42:29.744 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:42:29.744 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-25T23:42:29.744 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:42:29.744 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:42:29.744 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:42:29.744 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:42:29.744 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:42:29.745 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:42:29.745 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:42:29.745 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:42:29.745 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T23:42:29.745 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:42:29.745 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:42:29.745 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:42:29.745 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-25T23:42:29.745 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-25T23:42:29.745 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-25T23:42:29.745 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-25T23:42:29.745 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-25T23:42:29.745 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-25T23:42:29.745 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-25T23:42:29.745 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-25T23:42:29.746 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-25T23:42:29.746 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1 2024-03-25T23:42:29.746 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1 '!=' null 2024-03-25T23:42:30.500 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-03-25T23:42:30.500 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:42:30.500 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-25T23:42:30.500 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:42:30.500 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:42:30.500 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:42:30.500 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:42:30.500 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:42:30.500 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:42:30.500 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:42:30.500 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:42:30.500 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:42:30.501 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:42:30.501 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:42:30.501 INFO:tasks.workunit.client.0.smithi129.stderr:187400: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:42:30.501 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:42:30.501 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:42:30.501 INFO:tasks.workunit.client.0.smithi129.stderr:187400: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:42:30.501 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:42:30.518 INFO:tasks.workunit.client.0.smithi129.stderr:68: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:42:30.519 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T23:42:30.519 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:42:30.519 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:42:30.519 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:42:30.519 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-25T23:42:30.519 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-25T23:42:30.519 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-25T23:42:30.519 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-25T23:42:30.519 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-25T23:42:30.519 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-25T23:42:30.519 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-25T23:42:30.519 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-25T23:42:30.519 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-25T23:42:30.520 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1 2024-03-25T23:42:30.520 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1 '!=' null 2024-03-25T23:42:30.520 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-03-25T23:42:30.520 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:42:30.520 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-25T23:42:30.520 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:42:30.520 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:42:30.520 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:42:30.520 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:42:30.520 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:42:30.520 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:42:30.520 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:42:30.520 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:42:30.521 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:42:30.521 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:42:30.521 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:42:30.916 INFO:tasks.workunit.client.0.smithi129.stderr:187397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:42:30.916 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:42:30.916 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:42:30.916 INFO:tasks.workunit.client.0.smithi129.stderr:187397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:42:30.916 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:42:30.916 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-25T23:42:30.916 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 187396 2024-03-25T23:42:30.916 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-25T23:42:30.917 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-25T23:42:30.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-25T23:42:30.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-03-25T23:42:30.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-25T23:42:30.918 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-03-25T23:42:30.918 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-25T23:42:30.918 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-25T23:42:30.918 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-25T23:42:30.918 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-25T23:42:31.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-03-25T23:42:31.226 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-03-25T23:42:31.226 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-25T23:42:31.226 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-25T23:42:31.226 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:42:31.226 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:42:31.226 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:42:31.226 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:42:31.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-25T23:41:57.395859+0000 2024-03-25T23:42:31.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-25T23:42:31.791 INFO:tasks.workunit.client.0.smithi129.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-25T23:42:31.807 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-25T23:41:57.395859+0000 2024-03-25T23:42:31.807 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-25T23:42:31.807 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-25T23:41:57.395859+0000 2024-03-25T23:42:31.807 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-25T23:42:31.807 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-25T23:42:31.808 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:42:31.808 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:42:31.808 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:42:31.808 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:42:31.809 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:42:31.809 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:42:32.151 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:41:57.395859+0000 '>' 2024-03-25T23:41:57.395859+0000 2024-03-25T23:42:32.151 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:42:33.152 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:42:33.152 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:42:33.153 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:42:33.153 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:42:33.153 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:42:33.154 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:42:33.154 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:42:33.472 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:41:57.395859+0000 '>' 2024-03-25T23:41:57.395859+0000 2024-03-25T23:42:33.472 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:42:34.473 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:42:34.473 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:42:34.474 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:42:34.474 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:42:34.474 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:42:34.475 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:42:34.475 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:42:34.794 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:41:57.395859+0000 '>' 2024-03-25T23:41:57.395859+0000 2024-03-25T23:42:34.794 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:42:35.795 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:42:35.795 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:42:35.796 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:42:35.796 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:42:35.796 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:42:35.797 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:42:35.797 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:42:36.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:41:57.395859+0000 '>' 2024-03-25T23:41:57.395859+0000 2024-03-25T23:42:36.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:42:37.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:42:37.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:42:37.140 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:42:37.140 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:42:37.140 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:42:37.140 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:42:37.140 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:42:37.451 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:41:57.395859+0000 '>' 2024-03-25T23:41:57.395859+0000 2024-03-25T23:42:37.451 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:42:38.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:42:38.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:42:38.453 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:42:38.453 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:42:38.453 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:42:38.454 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:42:38.454 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:42:38.775 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:42:32.511747+0000 '>' 2024-03-25T23:41:57.395859+0000 2024-03-25T23:42:38.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-25T23:42:38.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-03-25T23:42:38.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-25T23:42:38.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:42:38.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:42:38.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 190565"' 2024-03-25T23:42:38.776 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 190565' 2024-03-25T23:42:38.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-25T23:42:38.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:42:38.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:42:38.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-25T23:42:38.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 190567"' 2024-03-25T23:42:38.777 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 190567' 2024-03-25T23:42:38.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/190568: /' 2024-03-25T23:42:38.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-03-25T23:42:38.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 190565 190567' 2024-03-25T23:42:38.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-25T23:42:38.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-25T23:42:38.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 190565 2024-03-25T23:42:38.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-25T23:42:38.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/190570: /' 2024-03-25T23:42:39.608 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:42:39.608 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:42:39.608 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-25T23:42:39.608 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:42:39.608 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-25T23:42:39.608 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:42:39.608 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:42:39.608 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-25T23:42:39.608 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:42:39.609 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-25T23:42:39.609 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:42:39.609 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:42:39.609 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:42:39.609 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:42:39.609 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:42:39.609 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:42:39.609 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-25T23:42:39.609 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:42:39.609 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:42:39.609 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-25T23:42:39.609 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:42:39.609 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-25T23:42:39.609 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-03-25T23:42:39.610 INFO:tasks.workunit.client.0.smithi129.stderr:190568: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-03-25T23:42:39.610 INFO:tasks.workunit.client.0.smithi129.stderr:190568: _ 2024-03-25T23:42:39.610 INFO:tasks.workunit.client.0.smithi129.stderr:190568: hinfo_key 2024-03-25T23:42:39.610 INFO:tasks.workunit.client.0.smithi129.stderr:190568: snapset 2024-03-25T23:42:39.610 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-25T23:42:39.610 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:42:39.610 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:42:39.610 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-25T23:42:39.610 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:42:39.610 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-25T23:42:39.610 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:42:39.612 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:42:39.613 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:42:39.613 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:42:39.613 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-25T23:42:39.613 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-25T23:42:39.613 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:42:39.613 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:42:39.613 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:42:39.613 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:42:39.613 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:42:39.613 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:42:39.613 INFO:tasks.workunit.client.0.smithi129.stderr:190568: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:42:39.613 INFO:tasks.workunit.client.0.smithi129.stderr:190568: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:42:39.613 INFO:tasks.workunit.client.0.smithi129.stderr:190568: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:42:39.614 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:42:39.614 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:42:39.614 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:42:39.614 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:42:39.614 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:42:39.614 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:42:39.615 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:42:39.615 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:42:39.615 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:42:39.615 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:42:39.615 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:42:39.615 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-25T23:42:39.615 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-25T23:42:39.615 INFO:tasks.workunit.client.0.smithi129.stderr:190568: start osd.3 2024-03-25T23:42:39.624 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:42:39.624 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:42:39.624 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-25T23:42:39.624 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:42:39.624 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-25T23:42:39.625 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:42:39.625 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:42:39.625 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-25T23:42:39.625 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:42:39.625 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-25T23:42:39.625 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:42:39.625 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:42:39.625 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:42:39.626 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:42:39.626 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:42:39.626 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:42:39.626 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-25T23:42:39.626 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:42:39.626 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:42:39.626 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-25T23:42:39.626 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:42:39.626 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:42:39.627 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-25T23:42:39.627 INFO:tasks.workunit.client.0.smithi129.stderr:190570: _ 2024-03-25T23:42:39.627 INFO:tasks.workunit.client.0.smithi129.stderr:190570: hinfo_key 2024-03-25T23:42:39.627 INFO:tasks.workunit.client.0.smithi129.stderr:190570: snapset 2024-03-25T23:42:39.627 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-25T23:42:39.627 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:42:39.627 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:42:39.628 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-25T23:42:39.628 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:42:39.628 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:42:39.628 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:42:39.628 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:42:39.628 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:42:39.628 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:42:39.628 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-25T23:42:39.628 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-25T23:42:39.629 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:42:39.629 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:42:39.629 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:42:39.629 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:42:39.629 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:42:39.629 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:42:39.629 INFO:tasks.workunit.client.0.smithi129.stderr:190570: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:42:39.629 INFO:tasks.workunit.client.0.smithi129.stderr:190570: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:42:39.629 INFO:tasks.workunit.client.0.smithi129.stderr:190570: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:42:39.629 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:42:39.629 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:42:39.629 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:42:39.630 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:42:39.630 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:42:39.630 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:42:39.630 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:42:39.632 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:42:39.632 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:42:39.632 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:42:39.632 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:42:39.632 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-25T23:42:39.632 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-25T23:42:39.632 INFO:tasks.workunit.client.0.smithi129.stderr:190570: start osd.1 2024-03-25T23:42:44.233 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:42:44.233 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-25T23:42:44.233 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-25T23:42:44.233 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:42:44.233 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:42:44.234 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:42:44.234 INFO:tasks.workunit.client.0.smithi129.stderr:190568: 2024-03-25T23:42:39.658+0000 7f70597327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:42:44.234 INFO:tasks.workunit.client.0.smithi129.stderr:190568: 2024-03-25T23:42:39.846+0000 7f70597327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:42:44.234 INFO:tasks.workunit.client.0.smithi129.stderr:190568: 2024-03-25T23:42:39.850+0000 7f70597327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:42:44.234 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-25T23:42:44.234 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:42:44.234 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-25T23:42:44.234 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:42:44.234 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:42:44.234 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:42:44.234 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:42:44.234 INFO:tasks.workunit.client.0.smithi129.stderr:190568: 0 2024-03-25T23:42:44.234 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:42:44.235 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:42:44.235 INFO:tasks.workunit.client.0.smithi129.stderr:190568: 2024-03-25T23:42:40.170+0000 7f70597327c0 -1 Falling back to public interface 2024-03-25T23:42:44.235 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:42:44.235 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:42:44.235 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:42:44.235 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:42:44.235 INFO:tasks.workunit.client.0.smithi129.stderr:190568: 1 2024-03-25T23:42:44.235 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:42:44.235 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:42:44.235 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:42:44.235 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:42:44.235 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:42:44.235 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:42:44.236 INFO:tasks.workunit.client.0.smithi129.stderr:190568: 2 2024-03-25T23:42:44.236 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:42:44.236 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:42:44.236 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:42:44.236 INFO:tasks.workunit.client.0.smithi129.stderr:190568: 2024-03-25T23:42:43.706+0000 7f70597327c0 -1 osd.3 91 log_to_monitors true 2024-03-25T23:42:44.236 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:42:44.236 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:42:44.236 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:42:44.236 INFO:tasks.workunit.client.0.smithi129.stderr:190568: 3 2024-03-25T23:42:44.236 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:42:44.599 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperovery_ops 2024-03-25T23:42:44.599 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-25T23:42:44.599 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-25T23:42:44.599 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:42:44.599 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:42:44.599 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:42:44.600 INFO:tasks.workunit.client.0.smithi129.stderr:190570: 2024-03-25T23:42:39.654+0000 7fdd5dde97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:42:44.600 INFO:tasks.workunit.client.0.smithi129.stderr:190570: 2024-03-25T23:42:39.846+0000 7fdd5dde97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:42:44.600 INFO:tasks.workunit.client.0.smithi129.stderr:190570: 2024-03-25T23:42:39.850+0000 7fdd5dde97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:42:44.600 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-25T23:42:44.600 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:42:44.600 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T23:42:44.600 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:42:44.600 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:42:44.600 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:42:44.600 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:42:44.600 INFO:tasks.workunit.client.0.smithi129.stderr:190570: 0 2024-03-25T23:42:44.600 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:42:44.600 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:42:44.601 INFO:tasks.workunit.client.0.smithi129.stderr:190570: 2024-03-25T23:42:40.394+0000 7fdd5dde97c0 -1 Falling back to public interface 2024-03-25T23:42:44.601 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:42:44.601 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:42:44.601 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:42:44.601 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:42:44.601 INFO:tasks.workunit.client.0.smithi129.stderr:190570: 1 2024-03-25T23:42:44.601 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:42:44.601 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:42:44.601 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:42:44.601 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:42:44.601 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:42:44.601 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:42:44.601 INFO:tasks.workunit.client.0.smithi129.stderr:190570: 2 2024-03-25T23:42:44.602 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:42:44.602 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:42:44.602 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:42:44.602 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:42:44.602 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:42:44.602 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:42:44.602 INFO:tasks.workunit.client.0.smithi129.stderr:190570: 3 2024-03-25T23:42:44.602 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:42:44.602 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:42:44.602 INFO:tasks.workunit.client.0.smithi129.stderr:190570: 2024-03-25T23:42:44.270+0000 7fdd5dde97c0 -1 osd.1 91 log_to_monitors true 2024-03-25T23:42:45.786 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/cs.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:42:45.786 INFO:tasks.workunit.client.0.smithi129.stderr:190568: osd.3 up in weight 1 up_from 96 up_thru 89 down_at 92 last_clean_interval [88,91) [v2:127.0.0.1:6810/4124107676,v1:127.0.0.1:6811/4124107676] [v2:127.0.0.1:6812/4124107676,v1:127.0.0.1:6813/4124107676] exists,up 2de53dc4-8cd5-4295-9d23-e6a36c158ac1 2024-03-25T23:42:45.786 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:42:45.786 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:42:45.786 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:42:45.787 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:42:45.787 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:42:45.787 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:42:45.787 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:42:45.787 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:42:45.787 INFO:tasks.workunit.client.0.smithi129.stderr:190568: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:42:45.787 INFO:tasks.workunit.client.0.smithi129.stderr:190568: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:42:45.787 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:42:45.787 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:42:45.787 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:42:45.787 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:42:45.787 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:42:45.787 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:42:45.787 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:42:45.788 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:42:45.788 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:42:45.788 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:42:45.788 INFO:tasks.workunit.client.0.smithi129.stderr:190568: 1 2024-03-25T23:42:45.788 INFO:tasks.workunit.client.0.smithi129.stderr:190568: 2 2024-03-25T23:42:45.788 INFO:tasks.workunit.client.0.smithi129.stderr:190568: 3' 2024-03-25T23:42:45.788 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:42:45.788 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:42:45.788 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:42:45.788 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836548 2024-03-25T23:42:45.788 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836548 2024-03-25T23:42:45.788 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548' 2024-03-25T23:42:45.788 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:42:45.788 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:42:45.788 INFO:tasks.workunit.client.0.smithi129.stderr:190568: Error ENXIO: problem getting command descriptions from osd.1 2024-03-25T23:42:45.789 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq= 2024-03-25T23:42:45.789 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z '' 2024-03-25T23:42:45.789 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: continue 2024-03-25T23:42:48.217 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_peph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:42:48.217 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:42:48.217 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:42:48.217 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:42:48.217 INFO:tasks.workunit.client.0.smithi129.stderr:190570: 4 2024-03-25T23:42:48.217 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:42:48.217 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:42:48.218 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:42:48.218 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:42:48.218 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:42:48.218 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:42:48.218 INFO:tasks.workunit.client.0.smithi129.stderr:190570: 5 2024-03-25T23:42:48.218 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:42:48.218 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:42:48.218 INFO:tasks.workunit.client.0.smithi129.stderr:190570: osd.1 up in weight 1 up_from 99 up_thru 99 down_at 92 last_clean_interval [89,91) [v2:127.0.0.1:6826/2541430693,v1:127.0.0.1:6827/2541430693] [v2:127.0.0.1:6828/2541430693,v1:127.0.0.1:6829/2541430693] exists,up e1891468-d0f5-4d04-9e87-8f7a4ef38ae8 2024-03-25T23:42:48.218 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:42:48.218 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:42:48.218 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:42:48.218 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:42:48.218 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:42:48.218 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:42:48.219 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:42:48.219 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:42:48.219 INFO:tasks.workunit.client.0.smithi129.stderr:190570: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:42:48.219 INFO:tasks.workunit.client.0.smithi129.stderr:190570: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:42:48.219 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:42:48.219 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:42:48.219 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:42:48.219 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:42:48.219 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:42:48.219 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:42:48.219 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:42:48.219 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:42:48.220 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:42:48.220 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:42:48.220 INFO:tasks.workunit.client.0.smithi129.stderr:190570: 1 2024-03-25T23:42:48.220 INFO:tasks.workunit.client.0.smithi129.stderr:190570: 2 2024-03-25T23:42:48.220 INFO:tasks.workunit.client.0.smithi129.stderr:190570: 3' 2024-03-25T23:42:48.220 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:42:48.220 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:42:49.074 INFO:tasks.workunit.client.0.smithi129.stderr:19057g_stats: for osd in $ids 2024-03-25T23:42:49.074 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:42:49.074 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318285 2024-03-25T23:42:49.074 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318285 2024-03-25T23:42:49.074 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548 2-352187318285' 2024-03-25T23:42:49.074 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:42:49.074 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:42:49.074 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=412316860418 2024-03-25T23:42:49.074 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 412316860418 2024-03-25T23:42:49.074 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548 2-352187318285 3-412316860418' 2024-03-25T23:42:49.074 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:42:49.075 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836548 2024-03-25T23:42:49.075 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:42:49.075 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:42:49.075 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836548 2024-03-25T23:42:49.075 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:42:49.075 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836548 2024-03-25T23:42:49.075 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836548' 2024-03-25T23:42:49.075 INFO:tasks.workunit.client.0.smithi129.stderr:190568: waiting osd.0 seq 21474836548 2024-03-25T23:42:49.075 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:42:49.075 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836548 -lt 21474836548 2024-03-25T23:42:49.075 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:42:49.075 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-352187318285 2024-03-25T23:42:49.075 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:42:49.075 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:42:49.076 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-352187318285 2024-03-25T23:42:49.076 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:42:49.076 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318285 2024-03-25T23:42:49.076 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 352187318285' 2024-03-25T23:42:49.076 INFO:tasks.workunit.client.0.smithi129.stderr:190568: waiting osd.2 seq 352187318285 2024-03-25T23:42:49.076 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:42:49.076 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318284 -lt 352187318285 2024-03-25T23:42:49.076 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:42:49.076 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:42:49.076 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:42:49.076 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318285 -lt 352187318285 2024-03-25T23:42:50.480 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:42:50.480 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-412316860418 2024-03-25T23:42:50.480 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:42:50.481 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:42:50.481 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-412316860418 2024-03-25T23:42:50.481 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:42:50.481 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=412316860418 2024-03-25T23:42:50.481 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 412316860418' 2024-03-25T23:42:50.481 INFO:tasks.workunit.client.0.smithi129.stderr:190568: waiting osd.3 seq 412316860418 2024-03-25T23:42:50.481 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:42:50.481 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 412316860418 -lt 412316860418 2024-03-25T23:42:50.481 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:42:50.481 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:42:50.481 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:42:50.481 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:42:50.482 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:42:50.482 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:42:50.482 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:42:50.482 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:42:50.482 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:42:50.482 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:42:50.482 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:42:50.482 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:42:50.482 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:42:50.482 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:42:50.482 INFO:tasks.workunit.client.0.smithi129.stderr:190568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:42:50.483 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:42:50.483 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:42:50.483 INFO:tasks.workunit.client.0.smithi129.stderr:190568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:42:50.483 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:42:50.483 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-25T23:42:50.483 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 190567 2024-03-25T23:42:50.557 INFO:tasks.workunit.client.0.smithi129.stderr:0: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:42:50.557 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836550 2024-03-25T23:42:50.557 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836550 2024-03-25T23:42:50.557 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550' 2024-03-25T23:42:50.557 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:42:50.558 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:42:50.558 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=425201762306 2024-03-25T23:42:50.558 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 425201762306 2024-03-25T23:42:50.558 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-425201762306' 2024-03-25T23:42:50.558 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:42:50.558 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:42:50.558 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318287 2024-03-25T23:42:50.558 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318287 2024-03-25T23:42:50.558 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-425201762306 2-352187318287' 2024-03-25T23:42:50.558 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:42:50.558 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:42:50.559 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=412316860419 2024-03-25T23:42:50.559 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 412316860419 2024-03-25T23:42:50.559 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-425201762306 2-352187318287 3-412316860419' 2024-03-25T23:42:50.559 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:42:50.559 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836550 2024-03-25T23:42:50.559 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:42:50.559 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:42:50.559 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836550 2024-03-25T23:42:50.559 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:42:50.559 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836550 2024-03-25T23:42:50.559 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836550' 2024-03-25T23:42:50.559 INFO:tasks.workunit.client.0.smithi129.stderr:190570: waiting osd.0 seq 21474836550 2024-03-25T23:42:50.559 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:42:50.560 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836548 -lt 21474836550 2024-03-25T23:42:50.560 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:42:50.560 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:42:50.560 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:42:50.560 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836550 -lt 21474836550 2024-03-25T23:42:51.892 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:42:51.892 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-425201762306 2024-03-25T23:42:51.892 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:42:51.892 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:42:51.892 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-425201762306 2024-03-25T23:42:51.892 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:42:51.892 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=425201762306 2024-03-25T23:42:51.892 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 425201762306' 2024-03-25T23:42:51.892 INFO:tasks.workunit.client.0.smithi129.stderr:190570: waiting osd.1 seq 425201762306 2024-03-25T23:42:51.892 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:42:51.892 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 425201762306 -lt 425201762306 2024-03-25T23:42:51.893 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:42:51.893 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-352187318287 2024-03-25T23:42:51.893 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:42:51.893 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:42:51.893 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-352187318287 2024-03-25T23:42:51.893 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:42:51.893 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318287 2024-03-25T23:42:51.893 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 352187318287' 2024-03-25T23:42:51.893 INFO:tasks.workunit.client.0.smithi129.stderr:190570: waiting osd.2 seq 352187318287 2024-03-25T23:42:51.894 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:42:51.894 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318287 -lt 352187318287 2024-03-25T23:42:51.894 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:42:51.894 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-412316860419 2024-03-25T23:42:51.894 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:42:51.894 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:42:51.894 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-412316860419 2024-03-25T23:42:51.894 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:42:51.894 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=412316860419 2024-03-25T23:42:51.894 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 412316860419' 2024-03-25T23:42:51.894 INFO:tasks.workunit.client.0.smithi129.stderr:190570: waiting osd.3 seq 412316860419 2024-03-25T23:42:51.894 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:42:51.895 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 412316860420 -lt 412316860419 2024-03-25T23:42:51.895 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:42:51.895 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:42:51.895 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:42:52.580 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:42:52.580 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:42:52.580 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:42:52.580 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:42:52.580 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:42:52.580 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:42:52.580 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:42:52.580 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:42:52.580 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:42:52.581 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:42:52.581 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:42:52.581 INFO:tasks.workunit.client.0.smithi129.stderr:190570: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:42:52.581 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:42:52.581 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:42:52.581 INFO:tasks.workunit.client.0.smithi129.stderr:190570: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:42:52.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:42:52.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-25T23:42:52.581 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-25T23:42:52.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-25T23:42:52.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-03-25T23:42:52.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-25T23:42:52.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-25T23:42:52.628 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-25T23:42:52.629 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-25T23:42:52.629 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-25T23:42:52.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T23:42:52.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-25T23:42:52.630 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:42:52.630 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:42:52.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:42:52.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:42:52.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:42:52.941 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:42:52.941 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T23:42:52.942 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T23:42:52.942 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T23:42:52.945 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T23:42:52.945 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T23:42:52.945 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T23:42:52.946 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:42:52.946 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T23:42:52.947 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T23:42:52.947 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:42:52.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T23:42:52.950 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T23:42:52.950 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-25T23:42:52.987 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T23:42:52.987 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:42:52.987 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:42:52.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-25T23:42:52.989 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T23:42:52.990 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T23:42:52.990 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-25T23:42:52.990 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-25T23:42:52.990 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-25T23:42:52.990 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-25T23:42:52.990 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-25T23:42:52.990 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T23:42:52.990 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-25T23:42:52.990 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:42:52.990 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:42:52.990 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:42:52.990 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:42:52.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:42:52.992 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:42:52.993 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T23:42:52.994 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T23:42:52.994 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T23:42:52.995 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T23:42:52.995 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T23:42:52.996 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T23:42:52.997 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:42:52.997 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T23:42:52.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T23:42:52.998 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:42:52.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T23:42:53.000 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T23:42:53.000 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-25T23:42:53.001 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T23:42:53.002 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:42:53.002 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:42:53.002 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-25T23:42:53.003 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T23:42:53.003 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T23:42:53.003 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-25T23:42:53.004 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-25T23:42:53.004 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:42:53.004 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:42:53.004 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61636 2024-03-25T23:42:53.007 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-25T23:42:53.007 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-25T23:42:53.007 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-25T23:42:53.007 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-25T23:42:53.007 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_lrc_appends td/osd-scrub-repair 2024-03-25T23:42:53.008 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:842: TEST_corrupt_and_repair_lrc_appends: corrupt_and_repair_lrc td/osd-scrub-repair false 2024-03-25T23:42:53.008 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:825: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2024-03-25T23:42:53.008 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:826: corrupt_and_repair_lrc: local allow_overwrites=false 2024-03-25T23:42:53.008 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:827: corrupt_and_repair_lrc: local poolname=ecpool 2024-03-25T23:42:53.008 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:829: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2024-03-25T23:42:53.008 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-25T23:42:53.008 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-25T23:42:53.008 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-25T23:42:53.008 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-25T23:42:53.008 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-25T23:42:53.008 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-25T23:42:53.486 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-25T23:42:53.486 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:42:53.486 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:42:53.487 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:42:53.487 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:42:53.487 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:42:53.487 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:42:53.488 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-25T23:42:53.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-25T23:42:53.699 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-25T23:42:53.699 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T23:42:53.699 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T23:42:53.700 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-25T23:42:53.700 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-25T23:42:53.700 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T23:42:53.701 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T23:42:53.701 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T23:42:53.701 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T23:42:53.701 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:42:53.701 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:42:53.702 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-25T23:42:53.702 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T23:42:53.702 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get fsid 2024-03-25T23:42:53.814 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-25T23:42:53.814 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T23:42:53.814 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T23:42:53.814 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-25T23:42:53.814 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-25T23:42:53.814 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T23:42:53.814 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T23:42:53.815 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T23:42:53.815 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T23:42:53.815 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:42:53.816 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:42:53.816 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-25T23:42:53.816 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T23:42:53.816 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get mon_host 2024-03-25T23:42:53.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:830: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2024-03-25T23:42:53.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-25T23:42:53.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-25T23:42:53.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-25T23:42:53.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-25T23:42:53.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-25T23:42:53.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-25T23:42:54.789 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-25T23:42:54.789 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:42:54.789 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:42:54.789 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:42:54.789 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:42:54.789 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:42:54.789 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:42:54.790 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T23:42:54.791 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T23:42:54.825 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: seq 0 9 2024-03-25T23:42:54.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-25T23:42:54.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 0 2024-03-25T23:42:54.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:42:54.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:42:54.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-25T23:42:54.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:42:54.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-25T23:42:54.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:42:54.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:42:54.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:42:54.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:42:54.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-25T23:42:54.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-25T23:42:54.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:42:54.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:42:54.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:42:54.830 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:42:54.830 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:42:54.830 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:42:54.830 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:42:54.830 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:42:54.831 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:42:54.831 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:42:54.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:42:54.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:42:54.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:42:54.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:42:54.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:42:54.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:42:54.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:42:54.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:42:54.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:42:54.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:42:54.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:42:54.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T23:42:54.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-25T23:42:54.832 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:42:54.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=61a8e94c-0977-4365-b095-2dc5e55e0e51 2024-03-25T23:42:54.832 INFO:tasks.workunit.client.0.smithi129.stdout:add osd0 61a8e94c-0977-4365-b095-2dc5e55e0e51 2024-03-25T23:42:54.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 61a8e94c-0977-4365-b095-2dc5e55e0e51' 2024-03-25T23:42:54.833 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:42:54.849 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD+CwJm3aTDMhAAAEtzIIN15VVlgkNB0Zh+qg== 2024-03-25T23:42:54.849 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD+CwJm3aTDMhAAAEtzIIN15VVlgkNB0Zh+qg=="}' 2024-03-25T23:42:54.849 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 61a8e94c-0977-4365-b095-2dc5e55e0e51 -i td/osd-scrub-repair/0/new.json 2024-03-25T23:42:55.126 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:42:55.141 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-25T23:42:55.143 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD+CwJm3aTDMhAAAEtzIIN15VVlgkNB0Zh+qg== --osd-uuid 61a8e94c-0977-4365-b095-2dc5e55e0e51 2024-03-25T23:42:55.171 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:42:55.170+0000 7f5b2632f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:42:55.194 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:42:55.190+0000 7f5b2632f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:42:55.195 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:42:55.194+0000 7f5b2632f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:42:55.195 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:42:55.194+0000 7f5b2632f7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-25T23:42:55.195 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:42:55.194+0000 7f5b2632f7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-25T23:42:55.195 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:42:55.194+0000 7f5b2632f7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-25T23:42:55.293 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:42:55.290+0000 7f5b2632f7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-25T23:43:01.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-25T23:43:01.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:43:01.569 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd0 key to auth repository 2024-03-25T23:43:01.569 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-25T23:43:01.569 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:43:01.957 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.0 2024-03-25T23:43:01.958 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-25T23:43:01.958 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:43:01.958 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:43:01.958 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:43:01.958 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:43:01.989 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:43:01.986+0000 7ff0b0f9b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:43:01.991 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:43:01.990+0000 7ff0b0f9b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:43:01.992 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:43:01.990+0000 7ff0b0f9b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:43:02.292 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-25T23:43:02.292 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:43:02.292 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T23:43:02.292 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:43:02.292 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:43:02.292 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:43:02.292 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:43:02.293 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:43:02.293 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:43:02.293 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:43:02.541 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:43:02.538+0000 7ff0b0f9b7c0 -1 Falling back to public interface 2024-03-25T23:43:02.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:43:03.614 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:43:03.614 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:43:03.615 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:43:03.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:43:03.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:43:03.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:43:03.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:43:04.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:43:04.916 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:43:04.916 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:43:04.916 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:43:04.916 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:43:04.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:43:05.113 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:43:05.110+0000 7ff0b0f9b7c0 -1 osd.0 0 log_to_monitors true 2024-03-25T23:43:05.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:43:06.247 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:43:06.248 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:43:06.248 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:43:06.248 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:43:06.249 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:43:06.249 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:43:06.571 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:43:07.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:43:07.573 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:43:07.573 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:43:07.573 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:43:07.574 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:43:07.574 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:43:08.058 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:43:09.059 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:43:09.060 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:43:09.060 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:43:09.060 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:43:09.061 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:43:09.061 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:43:09.369 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:43:10.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:43:10.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:43:10.371 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:43:10.372 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:43:10.372 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:43:10.372 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:43:10.685 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:43:11.687 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:43:11.687 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:43:11.687 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:43:11.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:43:11.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:43:11.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:43:12.013 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:43:13.014 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:43:13.014 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:43:13.015 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:43:13.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:43:13.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:43:13.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:43:13.327 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:43:14.328 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:43:14.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:43:14.329 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:43:14.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:43:14.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:43:14.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:43:14.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:43:15.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:43:15.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:43:15.653 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:43:15.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:43:15.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:43:15.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:43:16.064 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:43:17.066 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:43:17.066 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:43:17.066 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:43:17.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:43:17.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:43:17.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:43:17.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:43:18.381 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:43:18.381 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:43:18.381 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:43:18.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:43:18.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:43:18.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:43:18.694 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:43:19.696 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:43:19.696 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:43:19.696 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:43:19.696 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:43:19.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:43:19.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:43:20.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:43:21.393 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:43:21.393 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:43:21.394 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:43:21.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:43:21.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:43:21.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:43:22.311 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:43:23.313 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:43:23.313 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:43:23.314 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:43:23.314 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:43:23.314 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:43:23.314 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:43:24.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:43:25.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:43:25.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:43:25.087 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:43:25.087 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:43:25.087 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:43:25.087 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:43:26.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:43:27.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:43:27.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:43:27.231 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:43:27.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:43:27.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:43:27.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:43:28.073 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:43:29.075 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:43:29.075 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:43:29.075 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:43:29.076 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:43:29.076 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:43:29.076 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:43:29.590 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:43:30.592 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:43:30.592 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:43:30.593 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:43:30.593 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:43:30.593 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:43:30.593 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:43:30.911 INFO:tasks.workunit.client.0.smithi129.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2004932085,v1:127.0.0.1:6803/2004932085] [v2:127.0.0.1:6804/2004932085,v1:127.0.0.1:6805/2004932085] exists,up 61a8e94c-0977-4365-b095-2dc5e55e0e51 2024-03-25T23:43:30.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:43:30.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:43:30.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:43:30.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-25T23:43:30.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 1 2024-03-25T23:43:30.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:43:30.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:43:30.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-25T23:43:30.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:43:30.916 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:43:30.916 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:43:30.916 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:43:30.916 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:43:30.916 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:43:30.916 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-25T23:43:30.916 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-25T23:43:30.916 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:43:30.916 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:43:30.916 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:43:30.916 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:43:30.916 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:43:30.916 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:43:30.916 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:43:30.917 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:43:30.917 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:43:30.917 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:43:30.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:43:30.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:43:30.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:43:30.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:43:30.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:43:30.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:43:30.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:43:30.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:43:30.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:43:30.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:43:30.918 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:43:30.918 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T23:43:30.918 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-25T23:43:30.918 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:43:30.918 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cb1e17d0-d054-43b2-a4bc-d88d5fb1c309 2024-03-25T23:43:30.918 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 cb1e17d0-d054-43b2-a4bc-d88d5fb1c309' 2024-03-25T23:43:30.918 INFO:tasks.workunit.client.0.smithi129.stdout:add osd1 cb1e17d0-d054-43b2-a4bc-d88d5fb1c309 2024-03-25T23:43:30.919 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:43:30.938 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAiDAJmv8IPOBAAb/a78p3QE5ECXo9thTy+4w== 2024-03-25T23:43:30.938 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAiDAJmv8IPOBAAb/a78p3QE5ECXo9thTy+4w=="}' 2024-03-25T23:43:30.938 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cb1e17d0-d054-43b2-a4bc-d88d5fb1c309 -i td/osd-scrub-repair/1/new.json 2024-03-25T23:43:31.280 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:43:31.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-25T23:43:31.301 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAiDAJmv8IPOBAAb/a78p3QE5ECXo9thTy+4w== --osd-uuid cb1e17d0-d054-43b2-a4bc-d88d5fb1c309 2024-03-25T23:43:31.335 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:43:31.337+0000 7fcc216837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:43:31.382 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:43:31.385+0000 7fcc216837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:43:31.384 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:43:31.385+0000 7fcc216837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:43:31.384 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:43:31.385+0000 7fcc216837c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-25T23:43:31.384 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:43:31.385+0000 7fcc216837c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-25T23:43:31.385 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:43:31.385+0000 7fcc216837c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-25T23:43:31.457 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:43:31.457+0000 7fcc216837c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-25T23:43:38.252 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-25T23:43:38.252 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:43:38.253 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd1 key to auth repository 2024-03-25T23:43:38.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-25T23:43:38.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:43:38.630 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.1 2024-03-25T23:43:38.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-25T23:43:38.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:43:38.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:43:38.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:43:38.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:43:38.663 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:43:38.665+0000 7fc1f40887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:43:38.665 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:43:38.665+0000 7fc1f40887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:43:38.666 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:43:38.669+0000 7fc1f40887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:43:38.961 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-25T23:43:38.961 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:43:38.961 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T23:43:38.961 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:43:38.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:43:38.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:43:38.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:43:38.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:43:38.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:43:38.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:43:39.221 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:43:39.221+0000 7fc1f40887c0 -1 Falling back to public interface 2024-03-25T23:43:39.274 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:43:40.275 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:43:40.275 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:43:40.275 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:43:40.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:43:40.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:43:40.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:43:40.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:43:41.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:43:41.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:43:41.587 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:43:41.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:43:41.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:43:41.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:43:41.921 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:43:41.921+0000 7fc1f40887c0 -1 osd.1 0 log_to_monitors true 2024-03-25T23:43:41.941 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:43:42.942 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:43:42.943 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:43:42.943 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:43:42.943 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:43:42.943 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:43:42.943 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:43:43.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:43:44.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:43:44.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:43:44.420 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:43:44.421 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:43:44.421 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:43:44.421 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:43:44.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:43:45.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:43:45.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:43:45.742 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:43:45.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:43:45.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:43:45.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:43:46.320 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:43:47.322 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:43:47.322 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:43:47.323 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:43:47.323 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:43:47.323 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:43:47.323 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:43:47.733 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:43:48.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:43:48.736 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:43:48.736 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:43:48.736 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:43:48.736 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:43:48.737 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:43:49.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:43:50.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:43:50.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:43:50.140 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:43:50.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:43:50.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:43:50.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:43:50.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:43:51.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:43:51.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:43:51.513 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:43:51.514 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:43:51.514 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:43:51.514 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:43:51.942 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:43:52.944 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:43:52.944 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:43:52.944 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:43:52.945 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:43:52.945 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:43:52.945 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:43:53.268 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:43:54.270 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:43:54.270 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:43:54.271 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:43:54.271 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:43:54.271 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:43:54.271 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:43:54.593 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:43:55.595 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:43:55.595 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:43:55.595 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:43:55.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:43:55.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:43:55.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:43:55.993 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:43:56.995 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:43:56.995 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:43:56.995 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:43:56.996 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:43:56.996 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:43:56.996 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:43:57.606 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:43:58.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:43:58.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:43:58.609 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:43:58.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:43:58.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:43:58.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:43:59.252 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:44:00.253 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:44:00.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:44:00.254 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:44:00.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:44:00.255 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:44:00.255 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:44:00.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:44:01.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:44:01.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:44:01.968 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:44:01.968 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:44:01.968 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:44:01.968 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:44:02.327 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:44:03.328 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:44:03.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:44:03.329 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:44:03.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:44:03.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:44:03.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:44:03.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:44:04.717 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:44:04.718 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:44:04.718 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:44:04.718 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:44:04.718 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:44:04.719 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:44:05.357 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:44:06.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:44:06.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:44:06.359 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:44:06.360 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:44:06.360 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:44:06.360 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:44:07.110 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:44:08.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:44:08.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:44:08.112 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-25T23:44:08.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-25T23:44:08.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:44:08.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:44:08.433 INFO:tasks.workunit.client.0.smithi129.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2500687739,v1:127.0.0.1:6811/2500687739] [v2:127.0.0.1:6812/2500687739,v1:127.0.0.1:6813/2500687739] exists,up cb1e17d0-d054-43b2-a4bc-d88d5fb1c309 2024-03-25T23:44:08.437 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:44:08.437 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:44:08.438 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:44:08.438 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-25T23:44:08.438 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 2 2024-03-25T23:44:08.438 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:44:08.438 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:44:08.438 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-25T23:44:08.438 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:44:08.438 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-25T23:44:08.438 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:44:08.438 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:44:08.438 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:44:08.439 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:44:08.439 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-25T23:44:08.439 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-25T23:44:08.439 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:44:08.439 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:44:08.439 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:44:08.439 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:44:08.439 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:44:08.439 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:44:08.439 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:44:08.439 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:44:08.439 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:44:08.439 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:44:08.440 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:44:08.440 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:44:08.440 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:44:08.440 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:44:08.440 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:44:08.440 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:44:08.440 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:44:08.440 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:44:08.440 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:44:08.440 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:44:08.440 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:44:08.441 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T23:44:08.441 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-25T23:44:08.441 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:44:08.441 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5dc40f33-8014-4c8a-a466-82d78ce3a8e8 2024-03-25T23:44:08.441 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 5dc40f33-8014-4c8a-a466-82d78ce3a8e8' 2024-03-25T23:44:08.442 INFO:tasks.workunit.client.0.smithi129.stdout:add osd2 5dc40f33-8014-4c8a-a466-82d78ce3a8e8 2024-03-25T23:44:08.442 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:44:08.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBIDAJmVtCgGxAA5GBh2dQHyR9GvNCtsxcDQw== 2024-03-25T23:44:08.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBIDAJmVtCgGxAA5GBh2dQHyR9GvNCtsxcDQw=="}' 2024-03-25T23:44:08.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5dc40f33-8014-4c8a-a466-82d78ce3a8e8 -i td/osd-scrub-repair/2/new.json 2024-03-25T23:44:08.809 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:44:08.825 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-25T23:44:08.826 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBIDAJmVtCgGxAA5GBh2dQHyR9GvNCtsxcDQw== --osd-uuid 5dc40f33-8014-4c8a-a466-82d78ce3a8e8 2024-03-25T23:44:08.861 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:44:08.861+0000 7f4c4aecc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:44:08.911 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:44:08.913+0000 7f4c4aecc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:44:08.912 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:44:08.913+0000 7f4c4aecc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:44:08.913 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:44:08.913+0000 7f4c4aecc7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-25T23:44:08.913 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:44:08.913+0000 7f4c4aecc7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-25T23:44:08.913 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:44:08.913+0000 7f4c4aecc7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-25T23:44:08.988 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:44:08.989+0000 7f4c4aecc7c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-25T23:44:15.662 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-25T23:44:15.663 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:44:15.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-25T23:44:15.664 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd2 key to auth repository 2024-03-25T23:44:15.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:44:16.053 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.2 2024-03-25T23:44:16.054 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-25T23:44:16.054 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:44:16.054 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:44:16.054 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:44:16.054 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:44:16.086 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:44:16.084+0000 7f0faf8d07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:44:16.088 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:44:16.088+0000 7f0faf8d07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:44:16.090 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:44:16.092+0000 7f0faf8d07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:44:16.365 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-25T23:44:16.365 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:44:16.365 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-25T23:44:16.365 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:44:16.366 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:44:16.366 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:44:16.366 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:44:16.366 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:44:16.366 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:44:16.366 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:44:16.648 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:44:16.648+0000 7f0faf8d07c0 -1 Falling back to public interface 2024-03-25T23:44:16.708 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:44:17.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:44:17.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:44:17.711 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:44:17.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:44:17.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:44:17.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:44:18.031 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:44:19.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:44:19.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:44:19.033 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:44:19.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:44:19.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:44:19.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:44:19.375 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:44:19.458 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:44:19.460+0000 7f0faf8d07c0 -1 osd.2 0 log_to_monitors true 2024-03-25T23:44:20.377 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:44:20.377 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:44:20.377 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:44:20.377 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:44:20.378 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:44:20.378 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:44:20.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:44:21.912 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:44:21.912 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:44:21.912 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:44:21.912 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:44:21.913 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:44:21.913 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:44:22.531 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:44:23.533 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:44:23.533 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:44:23.533 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:44:23.534 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:44:23.534 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:44:23.534 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:44:23.853 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:44:24.855 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:44:24.855 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:44:24.855 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:44:24.856 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:44:24.856 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:44:24.856 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:44:25.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:44:26.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:44:26.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:44:26.179 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:44:26.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:44:26.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:44:26.180 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:44:26.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:44:27.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:44:27.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:44:27.567 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:44:27.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:44:27.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:44:27.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:44:27.879 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:44:28.880 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:44:28.880 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:44:28.880 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:44:28.881 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:44:28.881 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:44:28.881 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:44:29.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:44:30.248 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:44:30.248 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:44:30.248 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:44:30.249 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:44:30.249 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:44:30.249 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:44:30.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:44:31.678 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:44:31.678 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:44:31.678 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:44:31.678 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:44:31.679 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:44:31.679 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:44:31.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:44:32.993 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:44:32.993 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:44:32.993 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:44:32.993 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:44:32.994 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:44:32.994 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:44:33.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:44:34.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:44:34.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:44:34.347 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:44:34.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:44:34.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:44:34.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:44:35.098 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:44:36.100 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:44:36.100 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:44:36.100 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:44:36.101 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:44:36.101 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:44:36.101 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:44:36.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:44:37.909 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:44:37.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:44:37.910 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:44:37.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:44:37.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:44:37.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:44:38.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:44:39.699 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:44:39.700 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:44:39.700 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:44:39.701 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:44:39.701 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:44:39.701 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:44:40.198 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:44:41.200 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:44:41.200 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:44:41.201 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:44:41.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:44:41.202 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:44:41.202 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:44:41.718 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:44:42.720 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:44:42.720 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:44:42.721 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:44:42.721 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:44:42.721 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:44:42.721 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:44:43.300 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:44:44.301 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:44:44.302 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:44:44.302 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:44:44.302 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:44:44.302 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:44:44.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:44:44.652 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:44:45.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:44:45.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:44:45.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-25T23:44:45.654 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-25T23:44:45.655 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:44:45.655 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:44:45.957 INFO:tasks.workunit.client.0.smithi129.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/554329908,v1:127.0.0.1:6819/554329908] [v2:127.0.0.1:6820/554329908,v1:127.0.0.1:6821/554329908] exists,up 5dc40f33-8014-4c8a-a466-82d78ce3a8e8 2024-03-25T23:44:45.958 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:44:45.959 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:44:45.959 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:44:45.959 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-25T23:44:45.959 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 3 2024-03-25T23:44:45.959 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:44:45.959 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:44:45.959 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-03-25T23:44:45.959 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:44:45.959 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-03-25T23:44:45.959 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:44:45.959 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:44:45.959 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:44:45.960 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:44:45.960 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-25T23:44:45.960 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-25T23:44:45.960 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:44:45.960 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:44:45.960 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:44:45.960 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:44:45.960 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:44:45.960 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:44:45.960 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:44:45.960 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:44:45.960 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:44:45.962 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:44:45.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:44:45.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:44:45.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:44:45.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:44:45.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:44:45.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:44:45.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:44:45.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:44:45.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:44:45.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:44:45.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:44:45.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T23:44:45.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-03-25T23:44:45.963 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:44:45.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=711eac53-c0bb-40a8-a592-40574bbc51ac 2024-03-25T23:44:45.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 711eac53-c0bb-40a8-a592-40574bbc51ac' 2024-03-25T23:44:45.964 INFO:tasks.workunit.client.0.smithi129.stdout:add osd3 711eac53-c0bb-40a8-a592-40574bbc51ac 2024-03-25T23:44:45.964 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:44:45.984 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBtDAJmYKPUOhAAR4FS+siDvDDO3XjIDzj/Cg== 2024-03-25T23:44:45.984 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBtDAJmYKPUOhAAR4FS+siDvDDO3XjIDzj/Cg=="}' 2024-03-25T23:44:45.984 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 711eac53-c0bb-40a8-a592-40574bbc51ac -i td/osd-scrub-repair/3/new.json 2024-03-25T23:44:46.367 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:44:46.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-03-25T23:44:46.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBtDAJmYKPUOhAAR4FS+siDvDDO3XjIDzj/Cg== --osd-uuid 711eac53-c0bb-40a8-a592-40574bbc51ac 2024-03-25T23:44:46.421 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:44:46.420+0000 7f50342fa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:44:46.469 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:44:46.468+0000 7f50342fa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:44:46.471 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:44:46.472+0000 7f50342fa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:44:46.471 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:44:46.472+0000 7f50342fa7c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-25T23:44:46.471 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:44:46.472+0000 7f50342fa7c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-25T23:44:46.471 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:44:46.472+0000 7f50342fa7c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-25T23:44:46.576 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:44:46.576+0000 7f50342fa7c0 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-03-25T23:44:53.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-03-25T23:44:53.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:44:53.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-03-25T23:44:53.498 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd3 key to auth repository 2024-03-25T23:44:53.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:44:53.877 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-03-25T23:44:53.878 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.3 2024-03-25T23:44:53.878 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:44:53.878 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:44:53.878 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:44:53.879 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:44:53.909 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:44:53.908+0000 7f4628d3f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:44:53.926 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:44:53.928+0000 7f4628d3f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:44:53.928 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:44:53.928+0000 7f4628d3f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:44:54.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-03-25T23:44:54.195 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:44:54.195 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-25T23:44:54.195 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:44:54.195 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:44:54.197 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:44:54.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:44:54.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:44:54.198 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:44:54.198 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:44:54.492 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:44:54.492+0000 7f4628d3f7c0 -1 Falling back to public interface 2024-03-25T23:44:54.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:44:55.534 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:44:55.534 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:44:55.534 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:44:55.534 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:44:55.535 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:44:55.535 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:44:55.855 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:44:56.857 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:44:56.857 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:44:56.857 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:44:56.858 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:44:56.858 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:44:56.858 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:44:57.054 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:44:57.056+0000 7f4628d3f7c0 -1 osd.3 0 log_to_monitors true 2024-03-25T23:44:57.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:44:58.184 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:44:58.184 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:44:58.185 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:44:58.185 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:44:58.185 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:44:58.185 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:44:58.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:44:59.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:44:59.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:44:59.509 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:44:59.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:44:59.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:44:59.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:44:59.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:45:00.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:45:00.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:45:00.834 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:45:00.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:45:00.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:45:00.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:45:01.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:45:02.190 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:45:02.191 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:45:02.191 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:45:02.191 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:45:02.192 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:45:02.192 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:45:02.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:45:03.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:45:03.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:45:03.580 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:45:03.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:45:03.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:45:03.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:45:04.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:45:05.011 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:45:05.012 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:45:05.012 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:45:05.013 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:45:05.013 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:45:05.014 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:45:05.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:45:06.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:45:06.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:45:06.332 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:45:06.333 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:45:06.333 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:45:06.333 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:45:06.657 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:45:07.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:45:07.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:45:07.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:45:07.660 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:45:07.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:45:07.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:45:08.020 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:45:09.022 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:45:09.022 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:45:09.022 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:45:09.022 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:45:09.024 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:45:09.024 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:45:09.398 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:45:10.400 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:45:10.400 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:45:10.400 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:45:10.400 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:45:10.401 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:45:10.401 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:45:10.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:45:11.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:45:11.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:45:11.712 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:45:11.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:45:11.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:45:11.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:45:12.480 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:45:13.482 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:45:13.483 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:45:13.483 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:45:13.483 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:45:13.483 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:45:13.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:45:14.491 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:45:15.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:45:15.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:45:15.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:45:15.493 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:45:15.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:45:15.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:45:16.242 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:45:17.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:45:17.244 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:45:17.244 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:45:17.244 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:45:17.244 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:45:17.244 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:45:17.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:45:18.666 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:45:18.666 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:45:18.667 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:45:18.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:45:18.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:45:18.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:45:19.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:45:20.184 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:45:20.184 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:45:20.185 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:45:20.185 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:45:20.185 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:45:20.185 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:45:20.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:45:21.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:45:21.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:45:21.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:45:21.863 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:45:21.864 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:45:21.864 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:45:22.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:45:23.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:45:23.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:45:23.348 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-25T23:45:23.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-25T23:45:23.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:45:23.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:45:23.720 INFO:tasks.workunit.client.0.smithi129.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/810831453,v1:127.0.0.1:6827/810831453] [v2:127.0.0.1:6828/810831453,v1:127.0.0.1:6829/810831453] exists,up 711eac53-c0bb-40a8-a592-40574bbc51ac 2024-03-25T23:45:23.723 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:45:23.724 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:45:23.724 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:45:23.724 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-25T23:45:23.724 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 4 2024-03-25T23:45:23.724 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:45:23.724 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:45:23.724 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-03-25T23:45:23.724 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:45:23.724 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/4 2024-03-25T23:45:23.724 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:45:23.724 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:45:23.725 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:45:23.725 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:45:23.725 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2024-03-25T23:45:23.725 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2024-03-25T23:45:23.725 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:45:23.725 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:45:23.725 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:45:23.725 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:45:23.725 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:45:23.725 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:45:23.725 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:45:23.725 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:45:23.725 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:45:23.726 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:45:23.726 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:45:23.726 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:45:23.726 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:45:23.726 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:45:23.726 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:45:23.726 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:45:23.726 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:45:23.726 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:45:23.726 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:45:23.726 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:45:23.726 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:45:23.726 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T23:45:23.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/4 2024-03-25T23:45:23.727 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:45:23.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=50b954cb-0eaf-4f6b-8d79-94af84c288ea 2024-03-25T23:45:23.728 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 50b954cb-0eaf-4f6b-8d79-94af84c288ea' 2024-03-25T23:45:23.728 INFO:tasks.workunit.client.0.smithi129.stdout:add osd4 50b954cb-0eaf-4f6b-8d79-94af84c288ea 2024-03-25T23:45:23.728 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:45:23.749 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCTDAJmrlTOLBAA7smLj02xr8qfTID9uzK1sw== 2024-03-25T23:45:23.749 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCTDAJmrlTOLBAA7smLj02xr8qfTID9uzK1sw=="}' 2024-03-25T23:45:23.749 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 50b954cb-0eaf-4f6b-8d79-94af84c288ea -i td/osd-scrub-repair/4/new.json 2024-03-25T23:45:24.115 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:45:24.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/4/new.json 2024-03-25T23:45:24.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCTDAJmrlTOLBAA7smLj02xr8qfTID9uzK1sw== --osd-uuid 50b954cb-0eaf-4f6b-8d79-94af84c288ea 2024-03-25T23:45:24.171 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:45:24.171+0000 7f8635fe17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:45:24.217 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:45:24.215+0000 7f8635fe17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:45:24.219 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:45:24.219+0000 7f8635fe17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:45:24.219 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:45:24.219+0000 7f8635fe17c0 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-03-25T23:45:24.219 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:45:24.219+0000 7f8635fe17c0 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-03-25T23:45:24.219 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:45:24.219+0000 7f8635fe17c0 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-03-25T23:45:24.295 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:45:24.295+0000 7f8635fe17c0 -1 bluestore(td/osd-scrub-repair/4) _read_fsid unparsable uuid 2024-03-25T23:45:31.039 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/4/keyring 2024-03-25T23:45:31.039 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:45:31.040 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd4 key to auth repository 2024-03-25T23:45:31.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-03-25T23:45:31.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:45:31.425 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.4 2024-03-25T23:45:31.426 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-03-25T23:45:31.426 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:45:31.426 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:45:31.426 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:45:31.426 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:45:31.456 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:45:31.455+0000 7fdcf5b9d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:45:31.459 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:45:31.459+0000 7fdcf5b9d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:45:31.460 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:45:31.459+0000 7fdcf5b9d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:45:31.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-03-25T23:45:31.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:45:31.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-03-25T23:45:31.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:45:31.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:45:31.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:45:31.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:45:31.781 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:45:31.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:45:31.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:45:32.012 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:45:32.011+0000 7fdcf5b9d7c0 -1 Falling back to public interface 2024-03-25T23:45:32.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:45:33.121 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:45:33.121 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:45:33.121 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:45:33.122 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:45:33.122 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:45:33.122 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:45:33.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:45:34.436 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:45:34.436 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:45:34.437 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:45:34.437 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:45:34.437 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:45:34.437 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:45:34.615 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:45:34.615+0000 7fdcf5b9d7c0 -1 osd.4 0 log_to_monitors true 2024-03-25T23:45:34.837 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:45:35.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:45:35.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:45:35.839 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:45:35.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:45:35.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:45:35.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:45:36.209 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:45:37.211 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:45:37.211 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:45:37.212 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:45:37.212 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:45:37.212 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:45:37.212 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:45:37.792 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:45:38.794 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:45:38.794 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:45:38.794 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:45:38.795 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:45:38.795 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:45:38.795 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:45:39.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:45:40.115 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:45:40.115 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:45:40.116 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:45:40.116 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:45:40.117 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:45:40.117 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:45:40.540 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:45:41.544 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:45:41.544 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:45:41.544 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:45:41.545 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:45:41.545 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:45:41.545 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:45:41.857 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:45:42.859 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:45:42.859 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:45:42.859 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:45:42.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:45:42.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:45:42.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:45:43.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:45:44.167 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:45:44.167 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:45:44.167 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:45:44.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:45:44.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:45:44.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:45:44.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:45:45.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:45:45.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:45:45.609 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:45:45.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:45:45.610 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:45:45.610 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:45:45.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:45:46.990 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:45:46.990 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:45:46.990 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:45:46.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:45:46.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:45:46.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:45:47.312 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:45:48.314 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:45:48.314 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:45:48.315 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:45:48.315 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:45:48.315 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:45:48.315 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:45:48.939 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:45:49.941 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:45:49.941 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:45:49.941 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:45:49.942 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:45:49.942 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:45:49.942 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:45:50.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:45:51.668 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:45:51.668 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:45:51.668 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:45:51.669 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:45:51.669 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:45:51.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:45:52.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:45:53.666 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:45:53.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:45:53.667 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:45:53.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:45:53.668 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:45:53.668 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:45:54.465 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:45:55.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:45:55.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:45:55.467 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:45:55.468 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:45:55.468 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:45:55.468 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:45:56.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:45:57.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:45:57.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:45:57.390 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:45:57.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:45:57.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:45:57.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:45:58.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:45:59.134 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:45:59.134 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:45:59.135 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:45:59.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:45:59.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:45:59.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:45:59.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:46:00.737 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:46:00.737 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:46:00.737 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:46:00.738 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:46:00.738 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:46:00.738 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:46:01.089 INFO:tasks.workunit.client.0.smithi129.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/105106117,v1:127.0.0.1:6835/105106117] [v2:127.0.0.1:6836/105106117,v1:127.0.0.1:6837/105106117] exists,up 50b954cb-0eaf-4f6b-8d79-94af84c288ea 2024-03-25T23:46:01.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:46:01.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:46:01.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:46:01.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-25T23:46:01.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 5 2024-03-25T23:46:01.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:46:01.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:46:01.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-03-25T23:46:01.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:46:01.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/5 2024-03-25T23:46:01.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:46:01.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:46:01.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:46:01.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:46:01.094 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-25T23:46:01.094 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-25T23:46:01.094 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:46:01.094 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:46:01.094 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:46:01.094 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:46:01.094 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:46:01.094 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:46:01.094 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:46:01.094 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:46:01.094 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:46:01.094 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:46:01.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:46:01.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:46:01.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:46:01.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:46:01.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:46:01.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:46:01.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:46:01.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:46:01.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:46:01.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:46:01.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:46:01.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T23:46:01.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/5 2024-03-25T23:46:01.096 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:46:01.096 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=35c1c964-6ad5-4a39-91cc-74a5625271e2 2024-03-25T23:46:01.096 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 35c1c964-6ad5-4a39-91cc-74a5625271e2' 2024-03-25T23:46:01.096 INFO:tasks.workunit.client.0.smithi129.stdout:add osd5 35c1c964-6ad5-4a39-91cc-74a5625271e2 2024-03-25T23:46:01.097 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:46:01.117 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC5DAJmTr0yBxAAzguntfb80KVN/ysxcQUxuQ== 2024-03-25T23:46:01.117 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC5DAJmTr0yBxAAzguntfb80KVN/ysxcQUxuQ=="}' 2024-03-25T23:46:01.117 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 35c1c964-6ad5-4a39-91cc-74a5625271e2 -i td/osd-scrub-repair/5/new.json 2024-03-25T23:46:01.488 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:46:01.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/5/new.json 2024-03-25T23:46:01.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC5DAJmTr0yBxAAzguntfb80KVN/ysxcQUxuQ== --osd-uuid 35c1c964-6ad5-4a39-91cc-74a5625271e2 2024-03-25T23:46:01.537 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:46:01.534+0000 7fd75cca47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:46:01.606 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:46:01.606+0000 7fd75cca47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:46:01.607 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:46:01.606+0000 7fd75cca47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:46:01.608 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:46:01.606+0000 7fd75cca47c0 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-03-25T23:46:01.608 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:46:01.606+0000 7fd75cca47c0 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-03-25T23:46:01.608 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:46:01.606+0000 7fd75cca47c0 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-03-25T23:46:01.671 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:46:01.670+0000 7fd75cca47c0 -1 bluestore(td/osd-scrub-repair/5) _read_fsid unparsable uuid 2024-03-25T23:46:08.101 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/5/keyring 2024-03-25T23:46:08.101 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:46:08.102 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd5 key to auth repository 2024-03-25T23:46:08.103 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-03-25T23:46:08.103 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:46:08.499 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.5 2024-03-25T23:46:08.500 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-03-25T23:46:08.500 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:46:08.500 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:46:08.500 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:46:08.501 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:46:08.532 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:46:08.530+0000 7f96261047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:46:08.566 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:46:08.566+0000 7f96261047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:46:08.567 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:46:08.566+0000 7f96261047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:46:08.846 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-03-25T23:46:08.846 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:46:08.846 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-25T23:46:08.846 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:46:08.846 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:46:08.846 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:46:08.847 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:46:08.847 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:46:08.847 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:46:08.847 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:46:09.124 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:46:09.122+0000 7f96261047c0 -1 Falling back to public interface 2024-03-25T23:46:09.181 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:46:10.183 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:46:10.183 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:46:10.183 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:46:10.184 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:46:10.184 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:46:10.184 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:46:10.502 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:46:11.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:46:11.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:46:11.504 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:46:11.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:46:11.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:46:11.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:46:11.826 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:46:11.852 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:46:11.850+0000 7f96261047c0 -1 osd.5 0 log_to_monitors true 2024-03-25T23:46:12.828 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:46:12.828 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:46:12.828 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:46:12.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:46:12.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:46:12.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:46:13.296 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:46:14.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:46:14.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:46:14.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:46:14.299 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:46:14.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:46:14.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:46:14.677 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:46:15.679 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:46:15.679 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:46:15.679 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:46:15.680 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:46:15.680 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:46:15.680 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:46:16.014 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:46:17.016 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:46:17.016 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:46:17.017 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:46:17.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:46:17.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:46:17.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:46:17.339 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:46:18.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:46:18.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:46:18.341 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:46:18.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:46:18.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:46:18.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:46:18.657 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:46:19.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:46:19.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:46:19.660 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:46:19.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:46:19.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:46:19.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:46:20.014 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:46:21.016 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:46:21.016 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:46:21.016 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:46:21.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:46:21.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:46:21.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:46:21.334 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:46:22.336 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:46:22.336 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:46:22.336 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:46:22.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:46:22.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:46:22.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:46:22.657 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:46:23.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:46:23.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:46:23.660 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:46:23.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:46:23.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:46:23.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:46:24.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:46:25.180 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:46:25.180 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:46:25.180 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:46:25.180 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:46:25.181 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:46:25.181 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:46:25.483 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:46:26.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:46:26.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:46:26.486 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:46:26.486 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:46:26.486 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:46:26.486 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:46:27.366 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:46:28.367 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:46:28.368 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:46:28.368 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:46:28.368 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:46:28.369 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:46:28.369 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:46:29.272 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:46:30.274 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:46:30.274 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:46:30.274 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:46:30.275 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:46:30.275 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:46:30.275 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:46:31.022 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:46:32.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:46:32.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:46:32.024 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:46:32.024 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:46:32.024 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:46:32.024 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:46:32.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:46:33.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:46:33.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:46:33.518 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:46:33.518 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:46:33.518 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:46:33.518 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:46:34.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:46:35.361 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:46:35.361 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:46:35.362 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:46:35.362 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:46:35.362 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:46:35.362 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:46:35.842 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:46:36.844 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:46:36.844 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:46:36.844 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:46:36.845 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:46:36.845 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:46:36.845 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:46:37.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:46:38.656 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:46:38.656 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:46:38.656 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-25T23:46:38.657 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-25T23:46:38.657 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:46:38.657 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:46:38.974 INFO:tasks.workunit.client.0.smithi129.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/4004345405,v1:127.0.0.1:6843/4004345405] [v2:127.0.0.1:6844/4004345405,v1:127.0.0.1:6845/4004345405] exists,up 35c1c964-6ad5-4a39-91cc-74a5625271e2 2024-03-25T23:46:38.976 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:46:38.976 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:46:38.976 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:46:38.976 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-25T23:46:38.976 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 6 2024-03-25T23:46:38.976 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:46:38.976 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:46:38.976 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-03-25T23:46:38.977 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:46:38.977 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/6 2024-03-25T23:46:38.977 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:46:38.977 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:46:38.977 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:46:38.977 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:46:38.977 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2024-03-25T23:46:38.977 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2024-03-25T23:46:38.977 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:46:38.977 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:46:38.977 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:46:38.977 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:46:38.977 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:46:38.977 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:46:38.977 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:46:38.977 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:46:38.978 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:46:38.978 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:46:38.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:46:38.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:46:38.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:46:38.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:46:38.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:46:38.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:46:38.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:46:38.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:46:38.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:46:38.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:46:38.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:46:38.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T23:46:38.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/6 2024-03-25T23:46:38.980 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:46:38.980 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4803a9b5-01de-4032-9393-6ac967b22aad 2024-03-25T23:46:38.980 INFO:tasks.workunit.client.0.smithi129.stdout:add osd6 4803a9b5-01de-4032-9393-6ac967b22aad 2024-03-25T23:46:38.980 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 4803a9b5-01de-4032-9393-6ac967b22aad' 2024-03-25T23:46:38.981 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:46:38.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDfDAJmcQofABAAcnLeKIGiuSuKQ36d8iMEJQ== 2024-03-25T23:46:38.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDfDAJmcQofABAAcnLeKIGiuSuKQ36d8iMEJQ=="}' 2024-03-25T23:46:38.999 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4803a9b5-01de-4032-9393-6ac967b22aad -i td/osd-scrub-repair/6/new.json 2024-03-25T23:46:39.378 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:46:39.398 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/6/new.json 2024-03-25T23:46:39.400 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDfDAJmcQofABAAcnLeKIGiuSuKQ36d8iMEJQ== --osd-uuid 4803a9b5-01de-4032-9393-6ac967b22aad 2024-03-25T23:46:39.435 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:46:39.438+0000 7f15fd6fa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:46:39.480 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:46:39.482+0000 7f15fd6fa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:46:39.481 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:46:39.486+0000 7f15fd6fa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:46:39.482 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:46:39.486+0000 7f15fd6fa7c0 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-03-25T23:46:39.482 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:46:39.486+0000 7f15fd6fa7c0 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-03-25T23:46:39.482 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:46:39.486+0000 7f15fd6fa7c0 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-03-25T23:46:39.567 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:46:39.570+0000 7f15fd6fa7c0 -1 bluestore(td/osd-scrub-repair/6) _read_fsid unparsable uuid 2024-03-25T23:46:45.890 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/6/keyring 2024-03-25T23:46:45.891 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:46:45.893 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd6 key to auth repository 2024-03-25T23:46:45.893 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-03-25T23:46:45.893 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:46:46.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-03-25T23:46:46.279 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.6 2024-03-25T23:46:46.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:46:46.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:46:46.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:46:46.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:46:46.308 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:46:46.309+0000 7fe4d11677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:46:46.310 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:46:46.313+0000 7fe4d11677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:46:46.311 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:46:46.313+0000 7fe4d11677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:46:46.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-03-25T23:46:46.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:46:46.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-03-25T23:46:46.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:46:46.635 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:46:46.635 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:46:46.635 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:46:46.635 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:46:46.636 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:46:46.636 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:46:46.892 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:46:46.893+0000 7fe4d11677c0 -1 Falling back to public interface 2024-03-25T23:46:46.955 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:46:47.957 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:46:47.957 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:46:47.957 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:46:47.958 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:46:47.958 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:46:47.958 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:46:48.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:46:49.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:46:49.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:46:49.263 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:46:49.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:46:49.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:46:49.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:46:49.370 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:46:49.373+0000 7fe4d11677c0 -1 osd.6 0 log_to_monitors true 2024-03-25T23:46:49.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:46:50.647 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:46:50.647 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:46:50.647 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:46:50.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:46:50.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:46:50.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:46:50.965 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:46:51.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:46:51.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:46:51.968 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:46:51.968 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:46:51.968 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:46:51.968 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:46:52.313 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:46:53.314 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:46:53.315 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:46:53.315 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:46:53.315 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:46:53.316 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:46:53.316 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:46:53.818 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:46:54.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:46:54.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:46:54.820 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:46:54.821 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:46:54.821 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:46:54.821 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:46:55.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:46:56.144 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:46:56.144 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:46:56.145 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:46:56.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:46:56.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:46:56.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:46:56.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:46:57.552 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:46:57.552 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:46:57.553 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:46:57.554 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:46:57.554 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:46:57.554 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:46:57.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:46:58.876 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:46:58.876 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:46:58.877 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:46:58.877 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:46:58.877 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:46:58.877 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:46:59.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:47:00.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:47:00.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:47:00.201 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:47:00.202 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:47:00.202 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:47:00.202 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:47:00.686 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:47:01.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:47:01.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:47:01.688 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:47:01.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:47:01.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:47:01.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:47:02.012 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:47:03.014 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:47:03.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:47:03.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:47:03.015 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:47:03.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:47:03.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:47:03.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:47:04.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:47:04.333 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:47:04.333 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:47:04.333 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:47:04.333 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:47:04.333 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:47:04.902 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:47:05.904 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:47:05.904 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:47:05.904 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:47:05.905 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:47:05.905 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:47:05.905 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:47:06.728 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:47:07.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:47:07.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:47:07.731 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:47:07.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:47:07.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:47:07.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:47:08.624 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:47:09.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:47:09.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:47:09.627 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:47:09.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:47:09.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:47:09.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:47:10.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:47:11.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:47:11.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:47:11.511 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:47:11.512 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:47:11.512 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:47:11.512 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:47:12.407 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:47:13.408 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:47:13.408 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:47:13.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:47:13.409 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:47:13.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:47:13.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:47:14.424 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:47:15.426 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:47:15.426 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:47:15.427 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:47:15.427 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:47:15.427 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:47:15.427 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:47:15.828 INFO:tasks.workunit.client.0.smithi129.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/441868506,v1:127.0.0.1:6851/441868506] [v2:127.0.0.1:6852/441868506,v1:127.0.0.1:6853/441868506] exists,up 4803a9b5-01de-4032-9393-6ac967b22aad 2024-03-25T23:47:15.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:47:15.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:47:15.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:47:15.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-25T23:47:15.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 7 2024-03-25T23:47:15.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:47:15.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:47:15.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-03-25T23:47:15.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:47:15.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/7 2024-03-25T23:47:15.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:47:15.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:47:15.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:47:15.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:47:15.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2024-03-25T23:47:15.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2024-03-25T23:47:15.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:47:15.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:47:15.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:47:15.833 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:47:15.833 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:47:15.833 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:47:15.834 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:47:15.834 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:47:15.834 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:47:15.834 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:47:15.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:47:15.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:47:15.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:47:15.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:47:15.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:47:15.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:47:15.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:47:15.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:47:15.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:47:15.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:47:15.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:47:15.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T23:47:15.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/7 2024-03-25T23:47:15.836 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:47:15.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=eefa5485-5c85-4ecb-9047-c21da805cbca 2024-03-25T23:47:15.836 INFO:tasks.workunit.client.0.smithi129.stdout:add osd7 eefa5485-5c85-4ecb-9047-c21da805cbca 2024-03-25T23:47:15.837 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 eefa5485-5c85-4ecb-9047-c21da805cbca' 2024-03-25T23:47:15.837 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:47:15.853 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQADDQJmPxwMMxAA5jpYOG/D0LhIHb4UkKWirg== 2024-03-25T23:47:15.853 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQADDQJmPxwMMxAA5jpYOG/D0LhIHb4UkKWirg=="}' 2024-03-25T23:47:15.853 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new eefa5485-5c85-4ecb-9047-c21da805cbca -i td/osd-scrub-repair/7/new.json 2024-03-25T23:47:16.246 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:47:16.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/7/new.json 2024-03-25T23:47:16.268 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQADDQJmPxwMMxAA5jpYOG/D0LhIHb4UkKWirg== --osd-uuid eefa5485-5c85-4ecb-9047-c21da805cbca 2024-03-25T23:47:16.304 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:47:16.305+0000 7fe1dbe7e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:47:16.330 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:47:16.333+0000 7fe1dbe7e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:47:16.332 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:47:16.333+0000 7fe1dbe7e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:47:16.333 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:47:16.333+0000 7fe1dbe7e7c0 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-03-25T23:47:16.333 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:47:16.333+0000 7fe1dbe7e7c0 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-03-25T23:47:16.333 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:47:16.333+0000 7fe1dbe7e7c0 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-03-25T23:47:16.402 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:47:16.405+0000 7fe1dbe7e7c0 -1 bluestore(td/osd-scrub-repair/7) _read_fsid unparsable uuid 2024-03-25T23:47:22.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/7/keyring 2024-03-25T23:47:22.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:47:22.781 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd7 key to auth repository 2024-03-25T23:47:22.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-03-25T23:47:22.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:47:23.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-03-25T23:47:23.178 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.7 2024-03-25T23:47:23.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:47:23.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:47:23.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:47:23.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:47:23.204 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:47:23.204+0000 7f2ea77cb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:47:23.206 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:47:23.208+0000 7f2ea77cb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:47:23.207 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:47:23.208+0000 7f2ea77cb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:47:23.514 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-03-25T23:47:23.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:47:23.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-03-25T23:47:23.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:47:23.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:47:23.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:47:23.516 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:47:23.516 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:47:23.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:47:23.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:47:23.735 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:47:23.736+0000 7f2ea77cb7c0 -1 Falling back to public interface 2024-03-25T23:47:23.857 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:47:24.858 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:47:24.858 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:47:24.859 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:47:24.859 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:47:24.859 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:47:24.859 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:47:25.163 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:47:26.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:47:26.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:47:26.165 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:47:26.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:47:26.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:47:26.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:47:26.273 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:47:26.272+0000 7f2ea77cb7c0 -1 osd.7 0 log_to_monitors true 2024-03-25T23:47:26.526 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:47:27.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:47:27.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:47:27.528 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:47:27.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:47:27.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:47:27.529 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:47:27.845 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:47:28.847 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:47:28.847 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:47:28.847 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:47:28.848 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:47:28.848 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:47:28.848 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:47:29.210 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:47:30.211 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:47:30.211 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:47:30.212 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:47:30.212 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:47:30.212 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:47:30.212 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:47:30.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:47:31.533 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:47:31.534 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:47:31.534 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:47:31.534 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:47:31.535 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:47:31.535 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:47:31.857 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:47:32.858 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:47:32.859 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:47:32.859 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:47:32.859 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:47:32.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:47:32.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:47:33.180 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:47:34.183 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:47:34.183 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:47:34.183 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:47:34.184 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:47:34.184 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:47:34.184 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:47:34.501 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:47:35.503 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:47:35.503 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:47:35.503 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:47:35.504 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:47:35.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:47:35.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:47:35.818 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:47:36.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:47:36.821 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:47:36.821 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:47:36.821 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:47:36.821 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:47:36.821 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:47:37.143 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:47:38.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:47:38.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:47:38.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:47:38.146 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:47:38.146 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:47:38.146 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:47:38.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:47:39.466 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:47:39.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:47:39.467 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:47:39.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:47:39.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:47:39.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:47:40.523 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:47:41.525 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:47:41.526 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:47:41.526 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:47:41.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:47:41.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:47:41.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:47:42.574 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:47:43.576 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:47:43.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:47:43.577 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:47:43.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:47:43.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:47:43.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:47:44.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:47:45.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:47:45.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:47:45.435 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:47:45.436 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:47:45.436 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:47:45.436 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:47:46.324 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:47:47.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:47:47.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:47:47.326 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:47:47.327 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:47:47.327 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:47:47.327 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:47:47.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:47:48.718 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:47:48.718 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:47:48.718 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:47:48.719 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:47:48.719 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:47:48.719 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:47:49.381 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:47:50.383 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:47:50.383 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:47:50.384 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:47:50.384 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:47:50.384 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:47:50.384 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:47:50.733 INFO:tasks.workunit.client.0.smithi129.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/3133721756,v1:127.0.0.1:6859/3133721756] [v2:127.0.0.1:6860/3133721756,v1:127.0.0.1:6861/3133721756] exists,up eefa5485-5c85-4ecb-9047-c21da805cbca 2024-03-25T23:47:50.737 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:47:50.738 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:47:50.738 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:47:50.738 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-25T23:47:50.738 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 8 2024-03-25T23:47:50.738 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:47:50.738 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:47:50.738 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2024-03-25T23:47:50.738 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:47:50.738 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/8 2024-03-25T23:47:50.738 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:47:50.738 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:47:50.738 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:47:50.738 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:47:50.739 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2024-03-25T23:47:50.739 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2024-03-25T23:47:50.739 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:47:50.739 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:47:50.739 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:47:50.739 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:47:50.739 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:47:50.739 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:47:50.739 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:47:50.739 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:47:50.739 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:47:50.739 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:47:50.739 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:47:50.739 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:47:50.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:47:50.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:47:50.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:47:50.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:47:50.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:47:50.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:47:50.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:47:50.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:47:50.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:47:50.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T23:47:50.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/8 2024-03-25T23:47:50.741 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:47:50.741 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7d460d9a-d1a9-41c7-8946-6f2ec44f1004 2024-03-25T23:47:50.741 INFO:tasks.workunit.client.0.smithi129.stdout:add osd8 7d460d9a-d1a9-41c7-8946-6f2ec44f1004 2024-03-25T23:47:50.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 7d460d9a-d1a9-41c7-8946-6f2ec44f1004' 2024-03-25T23:47:50.742 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:47:50.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAmDQJmKthWLRAAyNBH8aXXkM3NSNKEZjcXkg== 2024-03-25T23:47:50.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAmDQJmKthWLRAAyNBH8aXXkM3NSNKEZjcXkg=="}' 2024-03-25T23:47:50.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7d460d9a-d1a9-41c7-8946-6f2ec44f1004 -i td/osd-scrub-repair/8/new.json 2024-03-25T23:47:51.149 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:47:51.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/8/new.json 2024-03-25T23:47:51.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAmDQJmKthWLRAAyNBH8aXXkM3NSNKEZjcXkg== --osd-uuid 7d460d9a-d1a9-41c7-8946-6f2ec44f1004 2024-03-25T23:47:51.204 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:47:51.204+0000 7f5106d2f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:47:51.242 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:47:51.244+0000 7f5106d2f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:47:51.244 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:47:51.244+0000 7f5106d2f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:47:51.245 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:47:51.244+0000 7f5106d2f7c0 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-03-25T23:47:51.245 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:47:51.244+0000 7f5106d2f7c0 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-03-25T23:47:51.245 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:47:51.244+0000 7f5106d2f7c0 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-03-25T23:47:51.319 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:47:51.320+0000 7f5106d2f7c0 -1 bluestore(td/osd-scrub-repair/8) _read_fsid unparsable uuid 2024-03-25T23:47:57.879 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/8/keyring 2024-03-25T23:47:57.879 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:47:57.880 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2024-03-25T23:47:57.881 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd8 key to auth repository 2024-03-25T23:47:57.881 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:47:58.302 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.8 2024-03-25T23:47:58.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2024-03-25T23:47:58.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:47:58.304 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:47:58.304 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:47:58.304 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:47:58.335 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:47:58.336+0000 7fbf780d77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:47:58.374 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:47:58.376+0000 7fbf780d77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:47:58.375 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:47:58.376+0000 7fbf780d77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:47:58.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2024-03-25T23:47:58.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:47:58.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2024-03-25T23:47:58.648 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:47:58.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:47:58.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:47:58.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:47:58.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:47:58.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:47:58.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:47:58.924 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:47:58.924+0000 7fbf780d77c0 -1 Falling back to public interface 2024-03-25T23:47:58.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:47:59.965 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:47:59.965 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:47:59.965 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:47:59.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:47:59.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:47:59.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:48:00.294 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:48:01.295 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:48:01.295 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:48:01.296 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:48:01.296 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:48:01.296 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:48:01.296 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:48:01.511 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:48:01.511+0000 7fbf780d77c0 -1 osd.8 0 log_to_monitors true 2024-03-25T23:48:01.622 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:48:02.624 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:48:02.624 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:48:02.624 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:48:02.625 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:48:02.625 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:48:02.625 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:48:02.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:48:03.950 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:48:03.950 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:48:03.951 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:48:03.951 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:48:03.951 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:48:03.951 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:48:04.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:48:05.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:48:05.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:48:05.309 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:48:05.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:48:05.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:48:05.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:48:05.728 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:48:06.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:48:06.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:48:06.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:48:06.730 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:48:06.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:48:06.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:48:07.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:48:08.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:48:08.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:48:08.037 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:48:08.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:48:08.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:48:08.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:48:08.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:48:09.486 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:48:09.486 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:48:09.487 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:48:09.487 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:48:09.487 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:48:09.487 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:48:09.794 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:48:10.796 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:48:10.796 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:48:10.797 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:48:10.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:48:10.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:48:10.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:48:11.114 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:48:12.116 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:48:12.116 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:48:12.116 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:48:12.117 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:48:12.117 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:48:12.117 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:48:12.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:48:13.512 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:48:13.512 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:48:13.512 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:48:13.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:48:13.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:48:13.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:48:13.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:48:14.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:48:14.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:48:14.835 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:48:14.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:48:14.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:48:14.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:48:15.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:48:16.656 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:48:16.656 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:48:16.656 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:48:16.657 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:48:16.657 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:48:16.657 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:48:17.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:48:18.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:48:18.125 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:48:18.125 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:48:18.125 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:48:18.125 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:48:18.126 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:48:18.895 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:48:19.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:48:19.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:48:19.897 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:48:19.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:48:19.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:48:19.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:48:20.614 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:48:21.616 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:48:21.616 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:48:21.617 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:48:21.617 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:48:21.617 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:48:21.617 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:48:22.087 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:48:23.088 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:48:23.089 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:48:23.089 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:48:23.089 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:48:23.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:48:23.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:48:23.533 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:48:24.535 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:48:24.535 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:48:24.535 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:48:24.536 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:48:24.536 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:48:24.536 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:48:24.859 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:48:25.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:48:25.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:48:25.861 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:48:25.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:48:25.862 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:48:25.862 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:48:26.223 INFO:tasks.workunit.client.0.smithi129.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/1845539889,v1:127.0.0.1:6867/1845539889] [v2:127.0.0.1:6868/1845539889,v1:127.0.0.1:6869/1845539889] exists,up 7d460d9a-d1a9-41c7-8946-6f2ec44f1004 2024-03-25T23:48:26.226 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:48:26.226 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:48:26.226 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:48:26.226 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-25T23:48:26.226 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 9 2024-03-25T23:48:26.226 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:48:26.226 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:48:26.226 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=9 2024-03-25T23:48:26.226 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:48:26.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/9 2024-03-25T23:48:26.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:48:26.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:48:26.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:48:26.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:48:26.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-25T23:48:26.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-25T23:48:26.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:48:26.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:48:26.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:48:26.227 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:48:26.227 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:48:26.227 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:48:26.227 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:48:26.228 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:48:26.228 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:48:26.228 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:48:26.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:48:26.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:48:26.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:48:26.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:48:26.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:48:26.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:48:26.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:48:26.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:48:26.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:48:26.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:48:26.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:48:26.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T23:48:26.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/9 2024-03-25T23:48:26.229 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:48:26.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e9f65c34-00ed-4379-b041-f45b4e8cc538 2024-03-25T23:48:26.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd9 e9f65c34-00ed-4379-b041-f45b4e8cc538' 2024-03-25T23:48:26.230 INFO:tasks.workunit.client.0.smithi129.stdout:add osd9 e9f65c34-00ed-4379-b041-f45b4e8cc538 2024-03-25T23:48:26.230 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:48:26.245 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBKDQJmtMzWDhAAyHB60lc6r6qEu5SF08WUpA== 2024-03-25T23:48:26.245 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBKDQJmtMzWDhAAyHB60lc6r6qEu5SF08WUpA=="}' 2024-03-25T23:48:26.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e9f65c34-00ed-4379-b041-f45b4e8cc538 -i td/osd-scrub-repair/9/new.json 2024-03-25T23:48:26.770 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:48:26.795 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/9/new.json 2024-03-25T23:48:26.796 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 9 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBKDQJmtMzWDhAAyHB60lc6r6qEu5SF08WUpA== --osd-uuid e9f65c34-00ed-4379-b041-f45b4e8cc538 2024-03-25T23:48:26.832 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:48:26.831+0000 7f925369c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:48:26.864 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:48:26.863+0000 7f925369c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:48:26.865 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:48:26.863+0000 7f925369c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:48:26.866 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:48:26.867+0000 7f925369c7c0 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-03-25T23:48:26.866 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:48:26.867+0000 7f925369c7c0 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-03-25T23:48:26.866 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:48:26.867+0000 7f925369c7c0 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-03-25T23:48:26.935 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:48:26.935+0000 7f925369c7c0 -1 bluestore(td/osd-scrub-repair/9) _read_fsid unparsable uuid 2024-03-25T23:48:33.073 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/9/keyring 2024-03-25T23:48:33.073 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:48:33.077 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd9 key to auth repository 2024-03-25T23:48:33.078 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd9 key to auth repository 2024-03-25T23:48:33.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:48:33.459 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.9 2024-03-25T23:48:33.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.9 2024-03-25T23:48:33.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 9 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:48:33.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:48:33.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:48:33.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:48:33.490 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:48:33.491+0000 7fc733e1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:48:33.493 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:48:33.491+0000 7fc733e1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:48:33.494 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:48:33.495+0000 7fc733e1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:48:33.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 9 2024-03-25T23:48:33.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:48:33.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-25T23:48:33.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:48:33.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:48:33.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:48:33.756 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:48:33.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:48:33.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:48:33.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:48:34.032 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:48:34.031+0000 7fc733e1d7c0 -1 Falling back to public interface 2024-03-25T23:48:34.098 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:48:35.098 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:48:35.099 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:48:35.099 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:48:35.099 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:48:35.100 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:48:35.100 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:48:35.417 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:48:36.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:48:36.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:48:36.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:48:36.421 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:48:36.421 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:48:36.421 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:48:36.505 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:48:36.503+0000 7fc733e1d7c0 -1 osd.9 0 log_to_monitors true 2024-03-25T23:48:36.739 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:48:37.741 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:48:37.741 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:48:37.742 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:48:37.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:48:37.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:48:37.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:48:38.068 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:48:39.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:48:39.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:48:39.070 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:48:39.071 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:48:39.071 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:48:39.071 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:48:39.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:48:40.393 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:48:40.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:48:40.394 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:48:40.395 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:48:40.395 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:48:40.396 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:48:40.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:48:41.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:48:41.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:48:41.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:48:41.833 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:48:41.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:48:41.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:48:42.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:48:43.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:48:43.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:48:43.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:48:43.231 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:48:43.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:48:43.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:48:43.551 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:48:44.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:48:44.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:48:44.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:48:44.553 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:48:44.554 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:48:44.554 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:48:44.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:48:45.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:48:45.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:48:45.872 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:48:45.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:48:45.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:48:45.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:48:46.244 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:48:47.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:48:47.247 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:48:47.247 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:48:47.247 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:48:47.247 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:48:47.247 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:48:47.573 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:48:48.575 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:48:48.575 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:48:48.576 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:48:48.576 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:48:48.576 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:48:48.576 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:48:48.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:48:49.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:48:49.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:48:49.899 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:48:49.900 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:48:49.900 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:48:49.900 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:48:50.969 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:48:51.971 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:48:51.971 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:48:51.971 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:48:51.971 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:48:51.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:48:51.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:48:52.673 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:48:53.675 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:48:53.675 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:48:53.676 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:48:53.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:48:53.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:48:53.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:48:54.063 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:48:55.065 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:48:55.065 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:48:55.065 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:48:55.065 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:48:55.066 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:48:55.066 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:48:55.475 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:48:56.476 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:48:56.477 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:48:56.477 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:48:56.477 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:48:56.477 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:48:56.477 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:48:57.128 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:48:58.129 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:48:58.129 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:48:58.129 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:48:58.130 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:48:58.130 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:48:58.130 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:48:58.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:48:59.745 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:48:59.745 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:48:59.745 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:48:59.745 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:48:59.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:48:59.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:49:00.066 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:49:01.068 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:49:01.068 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:49:01.068 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:49:01.069 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:49:01.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:49:01.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:49:01.394 INFO:tasks.workunit.client.0.smithi129.stdout:osd.9 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/1150562381,v1:127.0.0.1:6875/1150562381] [v2:127.0.0.1:6876/1150562381,v1:127.0.0.1:6877/1150562381] exists,up e9f65c34-00ed-4379-b041-f45b4e8cc538 2024-03-25T23:49:01.395 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:49:01.395 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:49:01.395 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:49:01.395 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: create_rbd_pool 2024-03-25T23:49:01.395 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-25T23:49:01.736 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'rbd' does not exist 2024-03-25T23:49:01.751 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-25T23:49:01.751 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-25T23:49:02.216 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'rbd' created 2024-03-25T23:49:02.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T23:49:03.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-25T23:49:05.672 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:835: corrupt_and_repair_lrc: wait_for_clean 2024-03-25T23:49:05.672 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:49:05.672 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:49:05.672 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:49:05.672 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:49:05.673 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:49:05.673 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:49:05.674 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:49:05.674 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:49:05.674 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:49:05.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:49:05.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:49:05.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:49:05.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:49:05.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:49:05.803 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:49:06.127 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:49:06.127 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:49:06.127 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:49:06.127 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:49:06.128 INFO:tasks.workunit.client.0.smithi129.stderr:4 2024-03-25T23:49:06.128 INFO:tasks.workunit.client.0.smithi129.stderr:5 2024-03-25T23:49:06.128 INFO:tasks.workunit.client.0.smithi129.stderr:6 2024-03-25T23:49:06.128 INFO:tasks.workunit.client.0.smithi129.stderr:7 2024-03-25T23:49:06.128 INFO:tasks.workunit.client.0.smithi129.stderr:8 2024-03-25T23:49:06.128 INFO:tasks.workunit.client.0.smithi129.stderr:9' 2024-03-25T23:49:06.128 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:49:06.128 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:49:06.128 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:49:06.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836557 2024-03-25T23:49:06.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836557 2024-03-25T23:49:06.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557' 2024-03-25T23:49:06.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:49:06.420 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:49:06.563 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673031 2024-03-25T23:49:06.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673031 2024-03-25T23:49:06.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673031' 2024-03-25T23:49:06.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:49:06.564 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:49:06.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509496 2024-03-25T23:49:06.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509496 2024-03-25T23:49:06.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673031 2-64424509496' 2024-03-25T23:49:06.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:49:06.705 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:49:06.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345975 2024-03-25T23:49:06.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345975 2024-03-25T23:49:06.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673031 2-64424509496 3-85899345975' 2024-03-25T23:49:06.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:49:06.838 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T23:49:06.978 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182446 2024-03-25T23:49:06.978 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182446 2024-03-25T23:49:07.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673031 2-64424509496 3-85899345975 4-107374182446' 2024-03-25T23:49:07.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:49:07.017 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T23:49:07.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018921 2024-03-25T23:49:07.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018921 2024-03-25T23:49:07.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673031 2-64424509496 3-85899345975 4-107374182446 5-128849018921' 2024-03-25T23:49:07.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:49:07.124 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T23:49:07.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855387 2024-03-25T23:49:07.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855387 2024-03-25T23:49:07.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673031 2-64424509496 3-85899345975 4-107374182446 5-128849018921 6-150323855387' 2024-03-25T23:49:07.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:49:07.257 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-25T23:49:07.395 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691860 2024-03-25T23:49:07.396 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691860 2024-03-25T23:49:07.396 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673031 2-64424509496 3-85899345975 4-107374182446 5-128849018921 6-150323855387 7-171798691860' 2024-03-25T23:49:07.396 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:49:07.396 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-25T23:49:07.538 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528333 2024-03-25T23:49:07.539 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528333 2024-03-25T23:49:07.539 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673031 2-64424509496 3-85899345975 4-107374182446 5-128849018921 6-150323855387 7-171798691860 8-193273528333' 2024-03-25T23:49:07.539 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:49:07.539 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-25T23:49:07.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364811 2024-03-25T23:49:07.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364811 2024-03-25T23:49:07.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673031 2-64424509496 3-85899345975 4-107374182446 5-128849018921 6-150323855387 7-171798691860 8-193273528333 9-214748364811' 2024-03-25T23:49:07.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:49:07.689 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836557 2024-03-25T23:49:07.689 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:49:07.690 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:49:07.691 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836557 2024-03-25T23:49:07.691 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:49:07.692 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836557 2024-03-25T23:49:07.692 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 21474836557 2024-03-25T23:49:07.692 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836557' 2024-03-25T23:49:07.693 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:49:07.999 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836557 -lt 21474836557 2024-03-25T23:49:07.999 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:49:08.000 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673031 2024-03-25T23:49:08.000 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:49:08.001 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:49:08.002 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673031 2024-03-25T23:49:08.002 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:49:08.003 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673031 2024-03-25T23:49:08.003 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673031' 2024-03-25T23:49:08.003 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 42949673031 2024-03-25T23:49:08.004 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:49:08.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673031 -lt 42949673031 2024-03-25T23:49:08.327 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:49:08.327 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:49:08.328 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509496 2024-03-25T23:49:08.328 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:49:08.329 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509496 2024-03-25T23:49:08.330 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:49:08.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509496 2024-03-25T23:49:08.331 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 64424509496 2024-03-25T23:49:08.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509496' 2024-03-25T23:49:08.331 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:49:08.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509496 -lt 64424509496 2024-03-25T23:49:08.668 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:49:08.668 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345975 2024-03-25T23:49:08.669 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:49:08.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:49:08.671 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345975 2024-03-25T23:49:08.671 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:49:08.672 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345975 2024-03-25T23:49:08.673 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345975' 2024-03-25T23:49:08.673 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.3 seq 85899345975 2024-03-25T23:49:08.673 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:49:09.011 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345975 -lt 85899345975 2024-03-25T23:49:09.011 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:49:09.012 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182446 2024-03-25T23:49:09.012 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:49:09.013 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T23:49:09.014 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182446 2024-03-25T23:49:09.014 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:49:09.016 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182446 2024-03-25T23:49:09.016 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.4 seq 107374182446 2024-03-25T23:49:09.016 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182446' 2024-03-25T23:49:09.016 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T23:49:09.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182446 -lt 107374182446 2024-03-25T23:49:09.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:49:09.333 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018921 2024-03-25T23:49:09.333 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:49:09.334 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T23:49:09.335 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018921 2024-03-25T23:49:09.335 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:49:09.336 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018921 2024-03-25T23:49:09.336 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018921' 2024-03-25T23:49:09.336 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.5 seq 128849018921 2024-03-25T23:49:09.337 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T23:49:09.672 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018922 -lt 128849018921 2024-03-25T23:49:09.672 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:49:09.673 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855387 2024-03-25T23:49:09.673 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:49:09.674 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-25T23:49:09.675 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855387 2024-03-25T23:49:09.675 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:49:09.678 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855387 2024-03-25T23:49:09.678 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855387' 2024-03-25T23:49:09.678 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.6 seq 150323855387 2024-03-25T23:49:09.678 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-25T23:49:10.057 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855387 -lt 150323855387 2024-03-25T23:49:10.058 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:49:10.058 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691860 2024-03-25T23:49:10.058 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:49:10.059 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-25T23:49:10.060 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691860 2024-03-25T23:49:10.061 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:49:10.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691860 2024-03-25T23:49:10.063 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.7 seq 171798691860 2024-03-25T23:49:10.063 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691860' 2024-03-25T23:49:10.063 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-25T23:49:10.399 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691860 -lt 171798691860 2024-03-25T23:49:10.399 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:49:10.400 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528333 2024-03-25T23:49:10.400 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:49:10.401 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-25T23:49:10.402 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528333 2024-03-25T23:49:10.402 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:49:10.403 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528333 2024-03-25T23:49:10.403 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528333' 2024-03-25T23:49:10.404 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.8 seq 193273528333 2024-03-25T23:49:10.404 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-25T23:49:10.728 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528333 -lt 193273528333 2024-03-25T23:49:10.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:49:10.729 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364811 2024-03-25T23:49:10.729 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:49:10.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-25T23:49:10.732 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364811 2024-03-25T23:49:10.732 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:49:10.733 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364811 2024-03-25T23:49:10.733 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364811' 2024-03-25T23:49:10.733 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.9 seq 214748364811 2024-03-25T23:49:10.733 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-25T23:49:11.050 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364811 -lt 214748364811 2024-03-25T23:49:11.050 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:49:11.051 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:49:11.051 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:49:11.432 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-25T23:49:11.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:49:11.433 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:49:11.434 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:49:11.434 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:49:11.434 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:49:11.434 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:49:11.434 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:49:11.787 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T23:49:11.787 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:49:11.788 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:49:11.788 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:49:12.152 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-25T23:49:12.152 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:49:12.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:49:12.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: create_ec_pool ecpool false k=4 m=2 l=3 plugin=lrc 2024-03-25T23:49:12.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-25T23:49:12.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-25T23:49:12.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-03-25T23:49:12.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-25T23:49:12.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2024-03-25T23:49:12.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-25T23:49:12.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-25T23:49:13.148 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'ecpool' created 2024-03-25T23:49:13.170 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T23:49:14.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-03-25T23:49:14.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-25T23:49:14.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:49:14.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:49:14.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:49:14.173 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:49:14.174 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:49:14.174 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:49:14.175 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:49:14.175 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:49:14.175 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:49:14.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:49:14.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:49:14.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:49:14.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:49:14.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:49:14.299 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:49:14.724 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:49:14.724 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:49:14.724 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:49:14.724 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:49:14.724 INFO:tasks.workunit.client.0.smithi129.stderr:4 2024-03-25T23:49:14.724 INFO:tasks.workunit.client.0.smithi129.stderr:5 2024-03-25T23:49:14.725 INFO:tasks.workunit.client.0.smithi129.stderr:6 2024-03-25T23:49:14.725 INFO:tasks.workunit.client.0.smithi129.stderr:7 2024-03-25T23:49:14.725 INFO:tasks.workunit.client.0.smithi129.stderr:8 2024-03-25T23:49:14.725 INFO:tasks.workunit.client.0.smithi129.stderr:9' 2024-03-25T23:49:14.725 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:49:14.725 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:49:14.725 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:49:15.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836559 2024-03-25T23:49:15.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836559 2024-03-25T23:49:15.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559' 2024-03-25T23:49:15.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:49:15.033 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:49:15.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673034 2024-03-25T23:49:15.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673034 2024-03-25T23:49:15.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673034' 2024-03-25T23:49:15.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:49:15.188 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:49:15.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509499 2024-03-25T23:49:15.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509499 2024-03-25T23:49:15.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673034 2-64424509499' 2024-03-25T23:49:15.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:49:15.391 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:49:15.538 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345978 2024-03-25T23:49:15.538 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345978 2024-03-25T23:49:15.538 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673034 2-64424509499 3-85899345978' 2024-03-25T23:49:15.538 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:49:15.538 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T23:49:15.680 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182448 2024-03-25T23:49:15.681 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182448 2024-03-25T23:49:15.681 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673034 2-64424509499 3-85899345978 4-107374182448' 2024-03-25T23:49:15.681 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:49:15.681 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T23:49:15.823 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018924 2024-03-25T23:49:15.823 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018924 2024-03-25T23:49:15.823 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673034 2-64424509499 3-85899345978 4-107374182448 5-128849018924' 2024-03-25T23:49:15.823 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:49:15.824 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T23:49:16.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855390 2024-03-25T23:49:16.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855390 2024-03-25T23:49:16.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673034 2-64424509499 3-85899345978 4-107374182448 5-128849018924 6-150323855390' 2024-03-25T23:49:16.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:49:16.047 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-25T23:49:16.362 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691863 2024-03-25T23:49:16.362 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691863 2024-03-25T23:49:16.362 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673034 2-64424509499 3-85899345978 4-107374182448 5-128849018924 6-150323855390 7-171798691863' 2024-03-25T23:49:16.362 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:49:16.362 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-25T23:49:16.530 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528336 2024-03-25T23:49:16.530 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528336 2024-03-25T23:49:16.530 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673034 2-64424509499 3-85899345978 4-107374182448 5-128849018924 6-150323855390 7-171798691863 8-193273528336' 2024-03-25T23:49:16.530 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:49:16.530 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-25T23:49:16.725 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364814 2024-03-25T23:49:16.725 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364814 2024-03-25T23:49:16.725 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673034 2-64424509499 3-85899345978 4-107374182448 5-128849018924 6-150323855390 7-171798691863 8-193273528336 9-214748364814' 2024-03-25T23:49:16.725 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:49:16.726 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836559 2024-03-25T23:49:16.726 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:49:16.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:49:16.727 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836559 2024-03-25T23:49:16.728 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:49:16.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836559 2024-03-25T23:49:16.729 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 21474836559 2024-03-25T23:49:16.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836559' 2024-03-25T23:49:16.729 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:49:17.039 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836559 -lt 21474836559 2024-03-25T23:49:17.039 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:49:17.040 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673034 2024-03-25T23:49:17.040 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:49:17.041 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:49:17.042 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673034 2024-03-25T23:49:17.042 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:49:17.043 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 42949673034 2024-03-25T23:49:17.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673034 2024-03-25T23:49:17.044 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673034' 2024-03-25T23:49:17.044 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:49:17.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673034 -lt 42949673034 2024-03-25T23:49:17.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:49:17.370 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509499 2024-03-25T23:49:17.370 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:49:17.372 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:49:17.373 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:49:17.373 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509499 2024-03-25T23:49:17.374 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509499 2024-03-25T23:49:17.374 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 64424509499 2024-03-25T23:49:17.375 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509499' 2024-03-25T23:49:17.375 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:49:17.826 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509499 -lt 64424509499 2024-03-25T23:49:17.826 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:49:17.827 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345978 2024-03-25T23:49:17.827 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:49:17.828 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:49:17.829 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345978 2024-03-25T23:49:17.829 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:49:17.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345978 2024-03-25T23:49:17.831 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.3 seq 85899345978 2024-03-25T23:49:17.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345978' 2024-03-25T23:49:17.831 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:49:18.155 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345978 -lt 85899345978 2024-03-25T23:49:18.156 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:49:18.156 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182448 2024-03-25T23:49:18.156 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:49:18.158 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T23:49:18.158 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182448 2024-03-25T23:49:18.159 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:49:18.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182448 2024-03-25T23:49:18.160 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.4 seq 107374182448 2024-03-25T23:49:18.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182448' 2024-03-25T23:49:18.160 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T23:49:18.490 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182449 -lt 107374182448 2024-03-25T23:49:18.490 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:49:18.490 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018924 2024-03-25T23:49:18.490 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:49:18.491 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T23:49:18.492 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:49:18.492 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018924 2024-03-25T23:49:18.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018924 2024-03-25T23:49:18.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018924' 2024-03-25T23:49:18.493 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.5 seq 128849018924 2024-03-25T23:49:18.494 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T23:49:18.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018924 -lt 128849018924 2024-03-25T23:49:18.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:49:18.835 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:49:18.835 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855390 2024-03-25T23:49:18.837 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-25T23:49:18.838 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855390 2024-03-25T23:49:18.838 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:49:18.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855390 2024-03-25T23:49:18.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855390' 2024-03-25T23:49:18.839 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.6 seq 150323855390 2024-03-25T23:49:18.839 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-25T23:49:19.167 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855390 -lt 150323855390 2024-03-25T23:49:19.167 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:49:19.168 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691863 2024-03-25T23:49:19.168 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:49:19.170 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-25T23:49:19.171 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691863 2024-03-25T23:49:19.171 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:49:19.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691863 2024-03-25T23:49:19.172 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.7 seq 171798691863 2024-03-25T23:49:19.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691863' 2024-03-25T23:49:19.173 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-25T23:49:19.487 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691863 -lt 171798691863 2024-03-25T23:49:19.487 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:49:19.487 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528336 2024-03-25T23:49:19.488 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:49:19.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-25T23:49:19.489 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528336 2024-03-25T23:49:19.490 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:49:19.491 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528336 2024-03-25T23:49:19.491 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528336' 2024-03-25T23:49:19.491 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.8 seq 193273528336 2024-03-25T23:49:19.491 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-25T23:49:19.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528336 -lt 193273528336 2024-03-25T23:49:19.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:49:19.803 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364814 2024-03-25T23:49:19.803 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:49:19.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-25T23:49:19.806 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:49:19.806 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364814 2024-03-25T23:49:19.807 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364814 2024-03-25T23:49:19.808 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.9 seq 214748364814 2024-03-25T23:49:19.808 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364814' 2024-03-25T23:49:19.808 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-25T23:49:20.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364814 -lt 214748364814 2024-03-25T23:49:20.124 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:49:20.125 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:49:20.125 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:49:20.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:49:20.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:49:20.482 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:49:20.482 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:49:20.482 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:49:20.482 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:49:20.482 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:49:20.482 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:49:20.794 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:49:20.794 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:49:20.795 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:49:20.795 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:49:21.147 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:49:21.147 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:49:21.147 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:49:21.148 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-25T23:49:21.148 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-03-25T23:49:21.148 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-03-25T23:49:21.148 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-03-25T23:49:21.148 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-03-25T23:49:21.148 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-25T23:49:21.148 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-25T23:49:21.148 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-03-25T23:49:21.148 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-25T23:49:21.148 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-25T23:49:21.148 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-25T23:49:21.622 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-25T23:49:21.643 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-25T23:49:22.249 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-25T23:49:22.269 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-25T23:49:22.270 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-25T23:49:22.270 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-25T23:49:22.775 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-03-25T23:49:22.775 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-03-25T23:49:22.775 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-25T23:49:22.775 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-03-25T23:49:22.776 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-25T23:49:23.099 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-03-25T23:49:23.101 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-03-25T23:49:23.101 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-03-25T23:49:23.101 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-25T23:49:23.101 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-03-25T23:49:23.101 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T23:49:23.102 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-03-25T23:49:23.431 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-25T23:49:23.431 INFO:tasks.workunit.client.0.smithi129.stderr:5 2024-03-25T23:49:23.431 INFO:tasks.workunit.client.0.smithi129.stderr:9 2024-03-25T23:49:23.431 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-25T23:49:23.431 INFO:tasks.workunit.client.0.smithi129.stderr:6 2024-03-25T23:49:23.431 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:49:23.431 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:49:23.431 INFO:tasks.workunit.client.0.smithi129.stderr:7' 2024-03-25T23:49:23.432 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 6 2 1 7 2024-03-25T23:49:23.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('5' '9' '0' '6' '2' '1' '7') 2024-03-25T23:49:23.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-03-25T23:49:23.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=5 2024-03-25T23:49:23.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=9 2024-03-25T23:49:23.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-03-25T23:49:23.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-25T23:49:23.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-03-25T23:49:23.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-03-25T23:49:23.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-25T23:49:23.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:49:23.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:49:23.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-25T23:49:23.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:49:23.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-03-25T23:49:23.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:49:23.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:49:23.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-25T23:49:23.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:49:23.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-25T23:49:23.435 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:49:23.435 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:49:23.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:49:23.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:49:23.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:49:23.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:49:23.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-03-25T23:49:23.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:49:23.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:49:23.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-25T23:49:23.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:49:23.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-25T23:49:23.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-03-25T23:49:25.723 INFO:tasks.workunit.client.0.smithi129.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-03-25T23:49:26.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-25T23:49:26.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:49:26.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:49:26.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-25T23:49:26.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:49:26.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-25T23:49:26.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:49:26.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:49:26.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:49:26.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:49:26.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-25T23:49:26.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-25T23:49:26.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:49:26.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:49:26.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:49:26.262 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:49:26.262 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:49:26.263 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:49:26.264 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:49:26.264 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:49:26.264 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:49:26.264 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:49:26.264 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:49:26.264 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:49:26.264 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:49:26.264 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:49:26.264 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:49:26.264 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:49:26.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:49:26.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:49:26.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:49:26.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:49:26.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-25T23:49:26.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-25T23:49:26.265 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.3 2024-03-25T23:49:26.266 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:49:26.266 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-25T23:49:26.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-25T23:49:26.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:49:26.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:49:26.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:49:26.299 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:49:26.301+0000 7f22cc66a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:49:26.302 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:49:26.305+0000 7f22cc66a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:49:26.304 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:49:26.305+0000 7f22cc66a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:49:26.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-25T23:49:26.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:49:26.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-25T23:49:26.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:49:26.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:49:26.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:49:26.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:49:26.586 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-25T23:49:26.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:49:26.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:49:26.844 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:49:26.845+0000 7f22cc66a7c0 -1 Falling back to public interface 2024-03-25T23:49:26.913 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:49:27.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:49:27.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:49:27.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:49:27.915 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:49:27.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:49:27.916 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:49:28.291 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:49:29.293 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:49:29.293 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:49:29.294 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:49:29.294 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:49:29.294 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:49:29.294 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:49:29.414 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:49:29.417+0000 7f22cc66a7c0 -1 osd.3 65 log_to_monitors true 2024-03-25T23:49:29.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:49:30.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:49:30.714 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:49:30.714 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:49:30.714 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:49:30.714 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:49:30.714 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:49:31.206 INFO:tasks.workunit.client.0.smithi129.stderr:osd.3 up in weight 1 up_from 69 up_thru 69 down_at 66 last_clean_interval [20,65) [v2:127.0.0.1:6826/1971690707,v1:127.0.0.1:6827/1971690707] [v2:127.0.0.1:6828/1971690707,v1:127.0.0.1:6829/1971690707] exists,up 711eac53-c0bb-40a8-a592-40574bbc51ac 2024-03-25T23:49:31.206 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:49:31.206 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:49:31.206 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:49:31.206 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:49:31.206 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:49:31.206 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:49:31.206 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:49:31.207 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:49:31.207 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:49:31.207 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:49:31.207 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:49:31.207 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:49:31.207 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:49:31.315 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:49:31.316 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:49:31.316 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:49:31.316 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:49:31.316 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:49:31.316 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:49:31.855 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:49:31.855 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:49:31.855 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:49:31.855 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:49:31.855 INFO:tasks.workunit.client.0.smithi129.stderr:4 2024-03-25T23:49:31.855 INFO:tasks.workunit.client.0.smithi129.stderr:5 2024-03-25T23:49:31.855 INFO:tasks.workunit.client.0.smithi129.stderr:6 2024-03-25T23:49:31.855 INFO:tasks.workunit.client.0.smithi129.stderr:7 2024-03-25T23:49:31.856 INFO:tasks.workunit.client.0.smithi129.stderr:8 2024-03-25T23:49:31.856 INFO:tasks.workunit.client.0.smithi129.stderr:9' 2024-03-25T23:49:31.856 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:49:31.856 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:49:31.856 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:49:32.016 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836564 2024-03-25T23:49:32.016 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836564 2024-03-25T23:49:32.016 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564' 2024-03-25T23:49:32.016 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:49:32.016 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:49:32.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673038 2024-03-25T23:49:32.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673038 2024-03-25T23:49:32.167 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673038' 2024-03-25T23:49:32.167 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:49:32.167 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:49:32.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509503 2024-03-25T23:49:32.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509503 2024-03-25T23:49:32.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673038 2-64424509503' 2024-03-25T23:49:32.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:49:32.310 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:49:32.455 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=296352743426 2024-03-25T23:49:32.455 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 296352743426 2024-03-25T23:49:32.455 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673038 2-64424509503 3-296352743426' 2024-03-25T23:49:32.455 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:49:32.455 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T23:49:32.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182453 2024-03-25T23:49:32.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182453 2024-03-25T23:49:32.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673038 2-64424509503 3-296352743426 4-107374182453' 2024-03-25T23:49:32.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:49:32.599 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T23:49:32.744 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018928 2024-03-25T23:49:32.744 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018928 2024-03-25T23:49:32.744 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673038 2-64424509503 3-296352743426 4-107374182453 5-128849018928' 2024-03-25T23:49:32.744 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:49:32.744 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T23:49:32.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855394 2024-03-25T23:49:32.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855394 2024-03-25T23:49:32.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673038 2-64424509503 3-296352743426 4-107374182453 5-128849018928 6-150323855394' 2024-03-25T23:49:32.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:49:32.887 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-25T23:49:33.031 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691867 2024-03-25T23:49:33.031 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691867 2024-03-25T23:49:33.031 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673038 2-64424509503 3-296352743426 4-107374182453 5-128849018928 6-150323855394 7-171798691867' 2024-03-25T23:49:33.031 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:49:33.031 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-25T23:49:33.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528340 2024-03-25T23:49:33.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528340 2024-03-25T23:49:33.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673038 2-64424509503 3-296352743426 4-107374182453 5-128849018928 6-150323855394 7-171798691867 8-193273528340' 2024-03-25T23:49:33.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:49:33.176 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-25T23:49:33.318 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364818 2024-03-25T23:49:33.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364818 2024-03-25T23:49:33.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673038 2-64424509503 3-296352743426 4-107374182453 5-128849018928 6-150323855394 7-171798691867 8-193273528340 9-214748364818' 2024-03-25T23:49:33.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:49:33.319 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836564 2024-03-25T23:49:33.319 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:49:33.320 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:49:33.321 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836564 2024-03-25T23:49:33.321 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:49:33.322 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836564 2024-03-25T23:49:33.322 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836564' 2024-03-25T23:49:33.323 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 21474836564 2024-03-25T23:49:33.323 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:49:33.622 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836564 -lt 21474836564 2024-03-25T23:49:33.622 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:49:33.623 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673038 2024-03-25T23:49:33.623 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:49:33.624 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:49:33.625 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673038 2024-03-25T23:49:33.625 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:49:33.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673038 2024-03-25T23:49:33.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673038' 2024-03-25T23:49:33.627 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 42949673038 2024-03-25T23:49:33.627 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:49:33.939 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673038 -lt 42949673038 2024-03-25T23:49:33.939 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:49:33.940 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509503 2024-03-25T23:49:33.940 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:49:33.942 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:49:33.943 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509503 2024-03-25T23:49:33.943 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:49:33.944 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509503 2024-03-25T23:49:33.944 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509503' 2024-03-25T23:49:33.944 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 64424509503 2024-03-25T23:49:33.944 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:49:34.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509503 -lt 64424509503 2024-03-25T23:49:34.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:49:34.262 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-296352743426 2024-03-25T23:49:34.262 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:49:34.264 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:49:34.264 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-296352743426 2024-03-25T23:49:34.265 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:49:34.266 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=296352743426 2024-03-25T23:49:34.266 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 296352743426' 2024-03-25T23:49:34.266 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.3 seq 296352743426 2024-03-25T23:49:34.266 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:49:34.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 296352743426 -lt 296352743426 2024-03-25T23:49:34.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:49:34.573 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182453 2024-03-25T23:49:34.573 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:49:34.575 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T23:49:34.575 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182453 2024-03-25T23:49:34.576 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:49:34.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182453 2024-03-25T23:49:34.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182453' 2024-03-25T23:49:34.577 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.4 seq 107374182453 2024-03-25T23:49:34.577 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T23:49:34.921 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182453 -lt 107374182453 2024-03-25T23:49:34.921 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:49:34.922 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018928 2024-03-25T23:49:34.922 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:49:34.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T23:49:34.924 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018928 2024-03-25T23:49:34.924 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:49:34.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018928 2024-03-25T23:49:34.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018928' 2024-03-25T23:49:34.926 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.5 seq 128849018928 2024-03-25T23:49:34.926 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T23:49:35.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018928 -lt 128849018928 2024-03-25T23:49:35.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:49:35.255 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855394 2024-03-25T23:49:35.255 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:49:35.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-25T23:49:35.257 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855394 2024-03-25T23:49:35.257 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:49:35.258 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855394 2024-03-25T23:49:35.258 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855394' 2024-03-25T23:49:35.259 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.6 seq 150323855394 2024-03-25T23:49:35.259 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-25T23:49:35.594 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855394 -lt 150323855394 2024-03-25T23:49:35.594 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:49:35.595 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691867 2024-03-25T23:49:35.595 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:49:35.597 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-25T23:49:35.598 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691867 2024-03-25T23:49:35.598 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:49:35.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691867 2024-03-25T23:49:35.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691867' 2024-03-25T23:49:35.599 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.7 seq 171798691867 2024-03-25T23:49:35.599 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-25T23:49:35.916 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691867 -lt 171798691867 2024-03-25T23:49:35.916 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:49:35.917 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528340 2024-03-25T23:49:35.917 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:49:35.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-25T23:49:35.920 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528340 2024-03-25T23:49:35.920 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:49:35.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528340 2024-03-25T23:49:35.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528340' 2024-03-25T23:49:35.922 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.8 seq 193273528340 2024-03-25T23:49:35.922 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-25T23:49:36.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528340 -lt 193273528340 2024-03-25T23:49:36.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:49:36.244 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364818 2024-03-25T23:49:36.244 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:49:36.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-25T23:49:36.247 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364818 2024-03-25T23:49:36.247 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:49:36.248 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364818 2024-03-25T23:49:36.248 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364818' 2024-03-25T23:49:36.248 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.9 seq 214748364818 2024-03-25T23:49:36.248 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-25T23:49:36.571 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364818 -lt 214748364818 2024-03-25T23:49:36.572 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:49:36.572 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:49:36.573 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:49:37.014 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:49:37.014 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:49:37.015 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:49:37.015 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:49:37.015 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:49:37.015 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:49:37.015 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:49:37.015 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:49:37.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:49:37.495 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:49:37.495 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:49:37.495 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:49:37.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:49:37.871 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:49:37.871 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:49:37.872 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-03-25T23:49:37.872 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-25T23:49:37.872 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-25T23:49:37.872 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-25T23:49:37.872 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-25T23:49:38.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-03-25T23:49:38.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-03-25T23:49:38.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-25T23:49:38.189 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-25T23:49:38.189 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:49:38.189 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:49:38.189 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:49:38.189 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:49:38.524 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-25T23:49:13.068301+0000 2024-03-25T23:49:38.525 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-25T23:49:38.815 INFO:tasks.workunit.client.0.smithi129.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-25T23:49:38.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-25T23:49:13.068301+0000 2024-03-25T23:49:38.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-25T23:49:38.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-25T23:49:13.068301+0000 2024-03-25T23:49:38.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-25T23:49:38.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-25T23:49:38.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:49:38.833 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:49:38.833 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:49:38.833 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:49:38.834 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:49:38.834 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:49:39.126 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:49:13.068301+0000 '>' 2024-03-25T23:49:13.068301+0000 2024-03-25T23:49:39.126 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:49:40.128 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:49:40.128 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:49:40.129 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:49:40.129 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:49:40.129 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:49:40.129 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:49:40.130 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:49:40.445 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:49:13.068301+0000 '>' 2024-03-25T23:49:13.068301+0000 2024-03-25T23:49:40.445 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:49:41.446 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:49:41.447 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:49:41.447 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:49:41.447 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:49:41.448 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:49:41.448 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:49:41.448 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:49:41.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:49:39.570727+0000 '>' 2024-03-25T23:49:13.068301+0000 2024-03-25T23:49:41.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-25T23:49:41.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-25T23:49:41.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:49:41.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:49:41.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-03-25T23:49:41.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:49:41.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-25T23:49:41.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:49:41.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:49:41.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-03-25T23:49:41.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:49:41.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-03-25T23:49:41.761 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:49:41.761 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:49:41.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:49:41.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:49:41.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:49:42.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:49:42.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-25T23:49:42.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:49:42.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:49:42.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-03-25T23:49:42.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:49:42.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-03-25T23:49:42.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-03-25T23:49:42.455 INFO:tasks.workunit.client.0.smithi129.stdout:_ 2024-03-25T23:49:42.455 INFO:tasks.workunit.client.0.smithi129.stdout:hinfo_key 2024-03-25T23:49:42.455 INFO:tasks.workunit.client.0.smithi129.stdout:snapset 2024-03-25T23:49:42.751 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-03-25T23:49:42.751 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:49:42.751 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:49:42.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-03-25T23:49:42.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:49:42.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-03-25T23:49:42.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:49:42.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:49:42.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:49:42.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:49:42.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-25T23:49:42.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-25T23:49:42.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:49:42.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:49:42.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:49:42.752 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:49:42.752 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:49:42.753 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:49:42.753 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:49:42.753 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:49:42.753 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:49:42.754 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:49:42.754 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:49:42.754 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:49:42.754 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:49:42.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:49:42.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:49:42.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:49:42.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:49:42.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:49:42.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:49:42.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:49:42.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-03-25T23:49:42.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-03-25T23:49:42.755 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.9 2024-03-25T23:49:42.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:49:42.756 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-03-25T23:49:42.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-03-25T23:49:42.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:49:42.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:49:42.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:49:42.789 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:49:42.789+0000 7fa0347467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:49:42.792 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:49:42.793+0000 7fa0347467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:49:42.794 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:49:42.797+0000 7fa0347467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:49:43.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-03-25T23:49:43.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:49:43.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-25T23:49:43.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:49:43.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:49:43.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:49:43.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:49:43.280 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-25T23:49:43.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:49:43.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:49:43.349 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:49:43.349+0000 7fa0347467c0 -1 Falling back to public interface 2024-03-25T23:49:43.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:49:44.636 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:49:44.636 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:49:44.636 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:49:44.636 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:49:44.637 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:49:44.637 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:49:44.959 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:49:45.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:49:45.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:49:45.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:49:45.963 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:49:45.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:49:45.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:49:46.312 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:49:46.343 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:49:46.345+0000 7fa0347467c0 -1 osd.9 70 log_to_monitors true 2024-03-25T23:49:47.314 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:49:47.314 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:49:47.314 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:49:47.315 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:49:47.315 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:49:47.315 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:49:47.735 INFO:tasks.workunit.client.0.smithi129.stderr:osd.9 up in weight 1 up_from 74 up_thru 0 down_at 71 last_clean_interval [50,70) [v2:127.0.0.1:6874/1835631314,v1:127.0.0.1:6875/1835631314] [v2:127.0.0.1:6876/1835631314,v1:127.0.0.1:6877/1835631314] exists,up e9f65c34-00ed-4379-b041-f45b4e8cc538 2024-03-25T23:49:47.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:49:47.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:49:47.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:49:47.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:49:47.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:49:47.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:49:47.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:49:47.735 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:49:47.736 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:49:47.736 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:49:47.736 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:49:47.736 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:49:47.736 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:49:47.852 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:49:47.852 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:49:47.852 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:49:47.852 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:49:47.852 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:49:47.852 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:49:48.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:49:48.230 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:49:48.230 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:49:48.230 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:49:48.230 INFO:tasks.workunit.client.0.smithi129.stderr:4 2024-03-25T23:49:48.230 INFO:tasks.workunit.client.0.smithi129.stderr:5 2024-03-25T23:49:48.230 INFO:tasks.workunit.client.0.smithi129.stderr:6 2024-03-25T23:49:48.231 INFO:tasks.workunit.client.0.smithi129.stderr:7 2024-03-25T23:49:48.231 INFO:tasks.workunit.client.0.smithi129.stderr:8 2024-03-25T23:49:48.231 INFO:tasks.workunit.client.0.smithi129.stderr:9' 2024-03-25T23:49:48.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:49:48.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:49:48.231 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:49:48.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836568 2024-03-25T23:49:48.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836568 2024-03-25T23:49:48.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568' 2024-03-25T23:49:48.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:49:48.394 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:49:48.679 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673043 2024-03-25T23:49:48.680 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673043 2024-03-25T23:49:48.680 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673043' 2024-03-25T23:49:48.680 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:49:48.680 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:49:48.827 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509507 2024-03-25T23:49:48.827 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509507 2024-03-25T23:49:48.827 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673043 2-64424509507' 2024-03-25T23:49:48.828 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:49:48.828 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:49:48.971 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=296352743430 2024-03-25T23:49:48.971 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 296352743430 2024-03-25T23:49:48.971 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673043 2-64424509507 3-296352743430' 2024-03-25T23:49:48.971 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:49:48.971 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T23:49:49.111 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182457 2024-03-25T23:49:49.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182457 2024-03-25T23:49:49.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673043 2-64424509507 3-296352743430 4-107374182457' 2024-03-25T23:49:49.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:49:49.112 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T23:49:49.255 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018933 2024-03-25T23:49:49.255 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018933 2024-03-25T23:49:49.255 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673043 2-64424509507 3-296352743430 4-107374182457 5-128849018933' 2024-03-25T23:49:49.255 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:49:49.255 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T23:49:49.400 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855398 2024-03-25T23:49:49.401 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855398 2024-03-25T23:49:49.401 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673043 2-64424509507 3-296352743430 4-107374182457 5-128849018933 6-150323855398' 2024-03-25T23:49:49.401 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:49:49.401 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-25T23:49:49.595 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691871 2024-03-25T23:49:49.595 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691871 2024-03-25T23:49:49.595 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673043 2-64424509507 3-296352743430 4-107374182457 5-128849018933 6-150323855398 7-171798691871' 2024-03-25T23:49:49.595 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:49:49.596 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-25T23:49:49.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528344 2024-03-25T23:49:49.747 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528344 2024-03-25T23:49:49.747 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673043 2-64424509507 3-296352743430 4-107374182457 5-128849018933 6-150323855398 7-171798691871 8-193273528344' 2024-03-25T23:49:49.747 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:49:49.747 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-25T23:49:49.889 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579906 2024-03-25T23:49:49.890 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579906 2024-03-25T23:49:49.890 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673043 2-64424509507 3-296352743430 4-107374182457 5-128849018933 6-150323855398 7-171798691871 8-193273528344 9-317827579906' 2024-03-25T23:49:49.890 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:49:49.890 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836568 2024-03-25T23:49:49.890 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:49:49.891 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:49:49.892 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836568 2024-03-25T23:49:49.892 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:49:49.893 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836568 2024-03-25T23:49:49.893 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836568' 2024-03-25T23:49:49.893 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 21474836568 2024-03-25T23:49:49.893 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:49:50.249 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836568 -lt 21474836568 2024-03-25T23:49:50.249 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:49:50.250 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673043 2024-03-25T23:49:50.251 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:49:50.252 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:49:50.253 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673043 2024-03-25T23:49:50.253 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:49:50.255 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673043 2024-03-25T23:49:50.255 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673043' 2024-03-25T23:49:50.255 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 42949673043 2024-03-25T23:49:50.255 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:49:50.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673043 -lt 42949673043 2024-03-25T23:49:50.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:49:50.587 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509507 2024-03-25T23:49:50.587 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:49:50.589 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:49:50.590 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509507 2024-03-25T23:49:50.590 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:49:50.592 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509507 2024-03-25T23:49:50.592 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509507' 2024-03-25T23:49:50.592 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 64424509507 2024-03-25T23:49:50.592 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:49:50.935 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509507 -lt 64424509507 2024-03-25T23:49:50.935 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:49:50.936 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-296352743430 2024-03-25T23:49:50.936 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:49:50.938 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:49:50.939 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-296352743430 2024-03-25T23:49:50.939 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:49:50.940 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=296352743430 2024-03-25T23:49:50.941 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 296352743430' 2024-03-25T23:49:50.941 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.3 seq 296352743430 2024-03-25T23:49:50.941 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:49:51.258 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 296352743430 -lt 296352743430 2024-03-25T23:49:51.258 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:49:51.259 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182457 2024-03-25T23:49:51.259 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:49:51.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T23:49:51.262 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:49:51.262 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182457 2024-03-25T23:49:51.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182457 2024-03-25T23:49:51.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182457' 2024-03-25T23:49:51.263 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.4 seq 107374182457 2024-03-25T23:49:51.264 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T23:49:51.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182458 -lt 107374182457 2024-03-25T23:49:51.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:49:51.623 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018933 2024-03-25T23:49:51.624 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:49:51.625 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T23:49:51.626 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018933 2024-03-25T23:49:51.626 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:49:51.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018933 2024-03-25T23:49:51.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018933' 2024-03-25T23:49:51.627 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.5 seq 128849018933 2024-03-25T23:49:51.627 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T23:49:51.993 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018933 -lt 128849018933 2024-03-25T23:49:51.994 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:49:51.994 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855398 2024-03-25T23:49:51.995 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:49:51.996 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-25T23:49:51.997 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855398 2024-03-25T23:49:51.997 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:49:51.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855398 2024-03-25T23:49:51.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855398' 2024-03-25T23:49:51.999 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.6 seq 150323855398 2024-03-25T23:49:51.999 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-25T23:49:52.316 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855399 -lt 150323855398 2024-03-25T23:49:52.316 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:49:52.317 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691871 2024-03-25T23:49:52.317 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:49:52.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-25T23:49:52.320 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691871 2024-03-25T23:49:52.320 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:49:52.321 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691871 2024-03-25T23:49:52.321 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691871' 2024-03-25T23:49:52.321 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.7 seq 171798691871 2024-03-25T23:49:52.321 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-25T23:49:52.642 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691872 -lt 171798691871 2024-03-25T23:49:52.642 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:49:52.643 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528344 2024-03-25T23:49:52.643 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:49:52.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-25T23:49:52.646 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528344 2024-03-25T23:49:52.646 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:49:52.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528344 2024-03-25T23:49:52.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528344' 2024-03-25T23:49:52.648 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.8 seq 193273528344 2024-03-25T23:49:52.648 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-25T23:49:52.975 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528345 -lt 193273528344 2024-03-25T23:49:52.975 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:49:52.976 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-317827579906 2024-03-25T23:49:52.976 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:49:52.978 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-25T23:49:52.978 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-317827579906 2024-03-25T23:49:52.979 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:49:52.980 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579906 2024-03-25T23:49:52.980 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 317827579906' 2024-03-25T23:49:52.980 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.9 seq 317827579906 2024-03-25T23:49:52.980 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-25T23:49:53.302 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579906 -lt 317827579906 2024-03-25T23:49:53.303 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:49:53.304 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:49:53.304 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:49:53.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:49:53.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:49:53.731 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:49:53.731 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:49:53.731 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:49:53.731 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:49:53.732 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:49:53.732 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:49:54.054 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:49:54.055 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:49:54.055 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:49:54.055 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:49:54.426 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:49:54.426 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:49:54.426 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:49:54.426 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-25T23:49:54.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-25T23:49:54.483 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2024-03-25T23:49:54.483 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-25T23:49:54.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-03-25T23:49:54.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=5 2024-03-25T23:49:54.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-25T23:49:54.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:49:54.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:49:54.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-25T23:49:54.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:49:54.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-03-25T23:49:54.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:49:54.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:49:54.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-25T23:49:54.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:49:54.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-25T23:49:54.484 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:49:54.485 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:49:54.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:49:54.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:49:54.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:49:54.792 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:49:54.792 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-03-25T23:49:54.792 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:49:54.792 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:49:54.792 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-25T23:49:54.792 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:49:54.792 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-25T23:49:54.792 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-03-25T23:49:57.011 INFO:tasks.workunit.client.0.smithi129.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-03-25T23:49:57.555 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-25T23:49:57.555 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:49:57.555 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:49:57.555 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-25T23:49:57.555 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:49:57.555 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-25T23:49:57.555 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:49:57.555 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:49:57.555 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:49:57.555 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:49:57.555 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-25T23:49:57.555 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-25T23:49:57.555 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:49:57.556 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:49:57.556 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:49:57.556 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:49:57.556 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:49:57.556 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:49:57.556 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:49:57.556 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:49:57.556 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:49:57.556 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:49:57.558 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:49:57.559 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:49:57.559 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:49:57.559 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:49:57.559 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:49:57.559 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:49:57.559 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:49:57.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:49:57.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:49:57.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:49:57.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-25T23:49:57.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-25T23:49:57.560 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.5 2024-03-25T23:49:57.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:49:57.561 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-25T23:49:57.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-25T23:49:57.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:49:57.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:49:57.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:49:57.592 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:49:57.593+0000 7f215a48b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:49:57.594 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:49:57.597+0000 7f215a48b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:49:57.596 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:49:57.597+0000 7f215a48b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:49:57.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-25T23:49:57.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:49:57.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-25T23:49:57.888 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:49:57.888 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:49:57.888 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:49:57.888 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:49:57.888 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-25T23:49:57.888 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:49:57.888 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:49:58.144 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:49:58.145+0000 7f215a48b7c0 -1 Falling back to public interface 2024-03-25T23:49:58.222 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:49:59.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:49:59.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:49:59.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:49:59.225 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:49:59.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:49:59.226 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:49:59.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:50:00.552 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:50:00.552 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:50:00.552 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:50:00.552 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:50:00.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:50:00.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:50:00.874 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:50:00.877+0000 7f215a48b7c0 -1 osd.5 75 log_to_monitors true 2024-03-25T23:50:00.880 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:50:01.882 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:50:01.882 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:50:01.882 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:50:01.882 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:50:01.882 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:50:01.882 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:50:02.280 INFO:tasks.workunit.client.0.smithi129.stderr:osd.5 up in weight 1 up_from 79 up_thru 79 down_at 76 last_clean_interval [30,75) [v2:127.0.0.1:6842/448061531,v1:127.0.0.1:6843/448061531] [v2:127.0.0.1:6844/448061531,v1:127.0.0.1:6845/448061531] exists,up 35c1c964-6ad5-4a39-91cc-74a5625271e2 2024-03-25T23:50:02.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:50:02.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:50:02.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:50:02.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:50:02.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:50:02.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:50:02.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:50:02.281 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:50:02.281 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:50:02.281 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:50:02.282 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:50:02.282 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:50:02.282 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:50:02.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:50:02.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:50:02.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:50:02.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:50:02.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:50:02.392 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:50:03.066 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:50:03.066 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:50:03.066 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:50:03.066 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:50:03.066 INFO:tasks.workunit.client.0.smithi129.stderr:4 2024-03-25T23:50:03.066 INFO:tasks.workunit.client.0.smithi129.stderr:5 2024-03-25T23:50:03.066 INFO:tasks.workunit.client.0.smithi129.stderr:6 2024-03-25T23:50:03.066 INFO:tasks.workunit.client.0.smithi129.stderr:7 2024-03-25T23:50:03.066 INFO:tasks.workunit.client.0.smithi129.stderr:8 2024-03-25T23:50:03.067 INFO:tasks.workunit.client.0.smithi129.stderr:9' 2024-03-25T23:50:03.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:50:03.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:50:03.067 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:50:03.223 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836572 2024-03-25T23:50:03.224 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836572 2024-03-25T23:50:03.224 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572' 2024-03-25T23:50:03.224 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:50:03.224 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:50:03.574 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673046 2024-03-25T23:50:03.574 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673046 2024-03-25T23:50:03.574 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673046' 2024-03-25T23:50:03.574 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:50:03.574 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:50:03.720 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509511 2024-03-25T23:50:03.720 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509511 2024-03-25T23:50:03.720 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673046 2-64424509511' 2024-03-25T23:50:03.720 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:50:03.720 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:50:03.866 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=296352743434 2024-03-25T23:50:03.866 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 296352743434 2024-03-25T23:50:03.866 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673046 2-64424509511 3-296352743434' 2024-03-25T23:50:03.866 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:50:03.866 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T23:50:04.014 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182461 2024-03-25T23:50:04.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182461 2024-03-25T23:50:04.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673046 2-64424509511 3-296352743434 4-107374182461' 2024-03-25T23:50:04.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:50:04.015 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T23:50:04.159 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=339302416386 2024-03-25T23:50:04.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 339302416386 2024-03-25T23:50:04.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673046 2-64424509511 3-296352743434 4-107374182461 5-339302416386' 2024-03-25T23:50:04.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:50:04.160 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T23:50:04.302 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855402 2024-03-25T23:50:04.302 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855402 2024-03-25T23:50:04.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673046 2-64424509511 3-296352743434 4-107374182461 5-339302416386 6-150323855402' 2024-03-25T23:50:04.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:50:04.303 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-25T23:50:04.444 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691875 2024-03-25T23:50:04.444 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691875 2024-03-25T23:50:04.444 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673046 2-64424509511 3-296352743434 4-107374182461 5-339302416386 6-150323855402 7-171798691875' 2024-03-25T23:50:04.444 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:50:04.444 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-25T23:50:04.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528348 2024-03-25T23:50:04.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528348 2024-03-25T23:50:04.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673046 2-64424509511 3-296352743434 4-107374182461 5-339302416386 6-150323855402 7-171798691875 8-193273528348' 2024-03-25T23:50:04.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:50:04.587 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-25T23:50:04.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579910 2024-03-25T23:50:04.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579910 2024-03-25T23:50:04.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673046 2-64424509511 3-296352743434 4-107374182461 5-339302416386 6-150323855402 7-171798691875 8-193273528348 9-317827579910' 2024-03-25T23:50:04.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:50:04.732 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:50:04.732 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836572 2024-03-25T23:50:04.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:50:04.732 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836572 2024-03-25T23:50:04.732 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:50:04.733 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836572 2024-03-25T23:50:04.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836572' 2024-03-25T23:50:04.734 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 21474836572 2024-03-25T23:50:04.734 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:50:05.041 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836572 -lt 21474836572 2024-03-25T23:50:05.041 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:50:05.042 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673046 2024-03-25T23:50:05.043 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:50:05.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:50:05.045 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:50:05.045 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673046 2024-03-25T23:50:05.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673046 2024-03-25T23:50:05.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673046' 2024-03-25T23:50:05.046 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 42949673046 2024-03-25T23:50:05.047 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:50:05.408 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673045 -lt 42949673046 2024-03-25T23:50:05.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:50:06.411 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:50:06.412 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:50:06.738 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673047 -lt 42949673046 2024-03-25T23:50:06.738 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:50:06.739 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509511 2024-03-25T23:50:06.739 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:50:06.741 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:50:06.742 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509511 2024-03-25T23:50:06.742 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:50:06.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509511 2024-03-25T23:50:06.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509511' 2024-03-25T23:50:06.743 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 64424509511 2024-03-25T23:50:06.744 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:50:07.060 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509512 -lt 64424509511 2024-03-25T23:50:07.060 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:50:07.060 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:50:07.061 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-296352743434 2024-03-25T23:50:07.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:50:07.062 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-296352743434 2024-03-25T23:50:07.063 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:50:07.064 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=296352743434 2024-03-25T23:50:07.064 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 296352743434' 2024-03-25T23:50:07.064 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.3 seq 296352743434 2024-03-25T23:50:07.064 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:50:07.447 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 296352743435 -lt 296352743434 2024-03-25T23:50:07.447 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:50:07.448 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182461 2024-03-25T23:50:07.448 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:50:07.450 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T23:50:07.451 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182461 2024-03-25T23:50:07.451 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:50:07.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182461 2024-03-25T23:50:07.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182461' 2024-03-25T23:50:07.452 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.4 seq 107374182461 2024-03-25T23:50:07.452 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T23:50:07.793 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182462 -lt 107374182461 2024-03-25T23:50:07.794 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:50:07.794 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-339302416386 2024-03-25T23:50:07.794 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:50:07.796 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T23:50:07.797 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-339302416386 2024-03-25T23:50:07.797 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:50:07.798 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=339302416386 2024-03-25T23:50:07.798 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 339302416386' 2024-03-25T23:50:07.798 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.5 seq 339302416386 2024-03-25T23:50:07.799 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T23:50:08.120 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 339302416387 -lt 339302416386 2024-03-25T23:50:08.120 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:50:08.121 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855402 2024-03-25T23:50:08.121 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:50:08.122 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-25T23:50:08.123 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855402 2024-03-25T23:50:08.123 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:50:08.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855402 2024-03-25T23:50:08.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855402' 2024-03-25T23:50:08.124 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.6 seq 150323855402 2024-03-25T23:50:08.125 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-25T23:50:08.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855403 -lt 150323855402 2024-03-25T23:50:08.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:50:08.436 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691875 2024-03-25T23:50:08.436 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:50:08.438 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-25T23:50:08.438 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:50:08.439 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691875 2024-03-25T23:50:08.440 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691875 2024-03-25T23:50:08.440 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691875' 2024-03-25T23:50:08.440 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.7 seq 171798691875 2024-03-25T23:50:08.440 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-25T23:50:08.769 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691876 -lt 171798691875 2024-03-25T23:50:08.769 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:50:08.770 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:50:08.770 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528348 2024-03-25T23:50:08.771 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-25T23:50:08.772 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528348 2024-03-25T23:50:08.772 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:50:08.774 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528348 2024-03-25T23:50:08.774 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528348' 2024-03-25T23:50:08.774 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.8 seq 193273528348 2024-03-25T23:50:08.775 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-25T23:50:09.091 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528349 -lt 193273528348 2024-03-25T23:50:09.091 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:50:09.092 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-317827579910 2024-03-25T23:50:09.092 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:50:09.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-25T23:50:09.094 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-317827579910 2024-03-25T23:50:09.094 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:50:09.096 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579910 2024-03-25T23:50:09.096 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 317827579910' 2024-03-25T23:50:09.096 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.9 seq 317827579910 2024-03-25T23:50:09.096 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-25T23:50:09.443 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579911 -lt 317827579910 2024-03-25T23:50:09.444 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:50:09.445 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:50:09.445 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:50:09.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:50:09.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:50:09.839 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:50:09.839 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:50:09.839 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:50:09.839 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:50:09.840 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:50:09.840 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:50:10.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:50:10.178 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:50:10.179 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:50:10.179 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:50:10.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:50:10.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:50:10.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:50:10.553 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-03-25T23:50:10.553 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-25T23:50:10.553 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-25T23:50:10.554 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-25T23:50:10.554 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-25T23:50:10.903 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-03-25T23:50:10.903 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-03-25T23:50:10.903 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-25T23:50:10.904 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-25T23:50:10.904 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:50:10.904 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:50:10.905 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:50:10.905 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:50:11.242 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-25T23:49:39.570727+0000 2024-03-25T23:50:11.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-25T23:50:11.537 INFO:tasks.workunit.client.0.smithi129.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-25T23:50:11.548 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-25T23:49:39.570727+0000 2024-03-25T23:50:11.549 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-25T23:50:11.549 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-25T23:49:39.570727+0000 2024-03-25T23:50:11.549 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-25T23:50:11.549 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-25T23:50:11.549 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:50:11.554 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:50:11.554 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:50:11.554 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:50:11.554 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:50:11.554 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:50:11.865 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:49:39.570727+0000 '>' 2024-03-25T23:49:39.570727+0000 2024-03-25T23:50:11.865 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:50:12.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:50:12.862 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:50:12.863 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:50:12.863 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:50:12.863 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:50:12.863 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:50:12.864 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:50:13.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:49:39.570727+0000 '>' 2024-03-25T23:49:39.570727+0000 2024-03-25T23:50:13.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:50:14.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:50:14.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:50:14.177 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:50:14.177 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:50:14.177 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:50:14.178 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:50:14.178 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:50:14.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:49:39.570727+0000 '>' 2024-03-25T23:49:39.570727+0000 2024-03-25T23:50:14.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:50:15.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:50:15.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:50:15.501 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:50:15.501 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:50:15.501 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:50:15.501 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:50:15.501 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:50:15.825 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:50:11.921237+0000 '>' 2024-03-25T23:49:39.570727+0000 2024-03-25T23:50:15.825 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-25T23:50:15.825 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-25T23:50:15.825 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:50:15.825 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:50:15.825 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-03-25T23:50:15.825 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:50:15.825 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-25T23:50:15.825 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:50:15.825 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:50:15.826 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-03-25T23:50:15.826 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:50:15.826 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-03-25T23:50:15.826 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:50:15.826 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:50:15.826 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:50:15.826 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:50:15.826 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:50:16.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:50:16.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-25T23:50:16.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:50:16.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:50:16.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-03-25T23:50:16.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:50:16.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-03-25T23:50:16.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-03-25T23:50:16.357 INFO:tasks.workunit.client.0.smithi129.stdout:_ 2024-03-25T23:50:16.357 INFO:tasks.workunit.client.0.smithi129.stdout:hinfo_key 2024-03-25T23:50:16.358 INFO:tasks.workunit.client.0.smithi129.stdout:snapset 2024-03-25T23:50:16.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-03-25T23:50:16.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:50:16.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:50:16.662 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-03-25T23:50:16.662 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:50:16.662 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-03-25T23:50:16.662 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:50:16.662 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:50:16.662 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:50:16.662 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:50:16.662 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-25T23:50:16.662 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-25T23:50:16.662 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:50:16.663 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:50:16.663 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:50:16.663 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:50:16.663 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:50:16.663 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:50:16.663 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:50:16.663 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:50:16.663 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:50:16.663 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:50:16.663 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:50:16.663 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:50:16.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:50:16.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:50:16.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:50:16.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:50:16.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:50:16.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:50:16.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:50:16.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:50:16.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-03-25T23:50:16.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-03-25T23:50:16.665 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.9 2024-03-25T23:50:16.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:50:16.665 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-03-25T23:50:16.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-03-25T23:50:16.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:50:16.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:50:16.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:50:16.692 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:50:16.692+0000 7f7240b3b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:50:16.726 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:50:16.728+0000 7f7240b3b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:50:16.728 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:50:16.728+0000 7f7240b3b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:50:16.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-03-25T23:50:16.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:50:16.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-25T23:50:16.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:50:16.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:50:16.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:50:16.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:50:16.998 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-25T23:50:16.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:50:16.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:50:17.273 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:50:17.272+0000 7f7240b3b7c0 -1 Falling back to public interface 2024-03-25T23:50:17.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:50:18.555 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:50:18.556 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:50:18.556 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:50:18.556 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:50:18.556 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:50:18.556 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:50:18.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:50:19.871 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:50:19.871 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:50:19.871 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:50:19.872 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:50:19.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:50:19.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:50:20.205 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:50:20.246 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:50:20.244+0000 7f7240b3b7c0 -1 osd.9 81 log_to_monitors true 2024-03-25T23:50:21.206 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:50:21.206 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:50:21.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:50:21.207 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:50:21.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:50:21.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:50:21.541 INFO:tasks.workunit.client.0.smithi129.stderr:osd.9 up in weight 1 up_from 85 up_thru 0 down_at 82 last_clean_interval [74,81) [v2:127.0.0.1:6874/744652870,v1:127.0.0.1:6875/744652870] [v2:127.0.0.1:6876/744652870,v1:127.0.0.1:6877/744652870] exists,up e9f65c34-00ed-4379-b041-f45b4e8cc538 2024-03-25T23:50:21.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:50:21.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:50:21.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:50:21.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:50:21.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:50:21.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:50:21.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:50:21.542 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:50:21.542 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:50:21.542 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:50:21.542 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:50:21.542 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:50:21.542 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:50:21.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:50:21.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:50:21.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:50:21.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:50:21.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:50:21.649 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:50:22.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:50:22.232 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:50:22.232 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:50:22.232 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:50:22.232 INFO:tasks.workunit.client.0.smithi129.stderr:4 2024-03-25T23:50:22.232 INFO:tasks.workunit.client.0.smithi129.stderr:5 2024-03-25T23:50:22.233 INFO:tasks.workunit.client.0.smithi129.stderr:6 2024-03-25T23:50:22.233 INFO:tasks.workunit.client.0.smithi129.stderr:7 2024-03-25T23:50:22.233 INFO:tasks.workunit.client.0.smithi129.stderr:8 2024-03-25T23:50:22.233 INFO:tasks.workunit.client.0.smithi129.stderr:9' 2024-03-25T23:50:22.233 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:50:22.233 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:50:22.233 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:50:22.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836577 2024-03-25T23:50:22.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836577 2024-03-25T23:50:22.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577' 2024-03-25T23:50:22.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:50:22.392 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:50:22.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673051 2024-03-25T23:50:22.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673051 2024-03-25T23:50:22.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673051' 2024-03-25T23:50:22.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:50:22.703 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:50:22.850 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509516 2024-03-25T23:50:22.850 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509516 2024-03-25T23:50:22.851 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509516' 2024-03-25T23:50:22.851 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:50:22.851 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:50:22.992 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=296352743439 2024-03-25T23:50:22.992 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 296352743439 2024-03-25T23:50:22.992 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509516 3-296352743439' 2024-03-25T23:50:22.992 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:50:22.992 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T23:50:23.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182466 2024-03-25T23:50:23.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182466 2024-03-25T23:50:23.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509516 3-296352743439 4-107374182466' 2024-03-25T23:50:23.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:50:23.136 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T23:50:23.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=339302416391 2024-03-25T23:50:23.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 339302416391 2024-03-25T23:50:23.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509516 3-296352743439 4-107374182466 5-339302416391' 2024-03-25T23:50:23.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:50:23.278 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T23:50:23.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855407 2024-03-25T23:50:23.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855407 2024-03-25T23:50:23.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509516 3-296352743439 4-107374182466 5-339302416391 6-150323855407' 2024-03-25T23:50:23.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:50:23.420 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-25T23:50:23.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691880 2024-03-25T23:50:23.554 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691880 2024-03-25T23:50:23.554 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509516 3-296352743439 4-107374182466 5-339302416391 6-150323855407 7-171798691880' 2024-03-25T23:50:23.554 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:50:23.554 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-25T23:50:23.692 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528353 2024-03-25T23:50:23.693 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528353 2024-03-25T23:50:23.693 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509516 3-296352743439 4-107374182466 5-339302416391 6-150323855407 7-171798691880 8-193273528353' 2024-03-25T23:50:23.693 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:50:23.693 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-25T23:50:23.825 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=365072220162 2024-03-25T23:50:23.825 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 365072220162 2024-03-25T23:50:23.825 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509516 3-296352743439 4-107374182466 5-339302416391 6-150323855407 7-171798691880 8-193273528353 9-365072220162' 2024-03-25T23:50:23.825 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:50:23.826 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836577 2024-03-25T23:50:23.826 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:50:23.826 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:50:23.827 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836577 2024-03-25T23:50:23.827 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:50:23.828 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836577 2024-03-25T23:50:23.828 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836577' 2024-03-25T23:50:23.828 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 21474836577 2024-03-25T23:50:23.829 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:50:24.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836577 -lt 21474836577 2024-03-25T23:50:24.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:50:24.133 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673051 2024-03-25T23:50:24.133 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:50:24.133 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:50:24.134 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673051 2024-03-25T23:50:24.134 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:50:24.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673051 2024-03-25T23:50:24.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673051' 2024-03-25T23:50:24.136 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 42949673051 2024-03-25T23:50:24.136 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:50:24.425 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673051 -lt 42949673051 2024-03-25T23:50:24.425 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:50:24.425 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509516 2024-03-25T23:50:24.426 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:50:24.428 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:50:24.428 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509516 2024-03-25T23:50:24.428 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:50:24.429 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509516 2024-03-25T23:50:24.430 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509516' 2024-03-25T23:50:24.430 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 64424509516 2024-03-25T23:50:24.430 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:50:24.785 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509516 -lt 64424509516 2024-03-25T23:50:24.785 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:50:24.786 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:50:24.787 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-296352743439 2024-03-25T23:50:24.787 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:50:24.788 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-296352743439 2024-03-25T23:50:24.789 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:50:24.790 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=296352743439 2024-03-25T23:50:24.790 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 296352743439' 2024-03-25T23:50:24.790 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.3 seq 296352743439 2024-03-25T23:50:24.790 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:50:25.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 296352743439 -lt 296352743439 2024-03-25T23:50:25.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:50:25.163 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182466 2024-03-25T23:50:25.164 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:50:25.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T23:50:25.166 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182466 2024-03-25T23:50:25.166 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:50:25.167 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182466 2024-03-25T23:50:25.167 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182466' 2024-03-25T23:50:25.167 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.4 seq 107374182466 2024-03-25T23:50:25.168 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T23:50:25.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182466 -lt 107374182466 2024-03-25T23:50:25.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:50:25.490 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-339302416391 2024-03-25T23:50:25.490 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:50:25.491 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T23:50:25.492 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-339302416391 2024-03-25T23:50:25.492 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:50:25.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=339302416391 2024-03-25T23:50:25.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 339302416391' 2024-03-25T23:50:25.494 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.5 seq 339302416391 2024-03-25T23:50:25.494 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T23:50:25.851 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 339302416391 -lt 339302416391 2024-03-25T23:50:25.851 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:50:25.851 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:50:25.851 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855407 2024-03-25T23:50:25.852 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-25T23:50:25.853 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855407 2024-03-25T23:50:25.853 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:50:25.855 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855407 2024-03-25T23:50:25.855 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855407' 2024-03-25T23:50:25.855 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.6 seq 150323855407 2024-03-25T23:50:25.855 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-25T23:50:26.181 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855407 -lt 150323855407 2024-03-25T23:50:26.181 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:50:26.182 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691880 2024-03-25T23:50:26.182 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:50:26.183 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-25T23:50:26.184 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691880 2024-03-25T23:50:26.184 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:50:26.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691880 2024-03-25T23:50:26.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691880' 2024-03-25T23:50:26.186 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.7 seq 171798691880 2024-03-25T23:50:26.186 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-25T23:50:26.521 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691880 -lt 171798691880 2024-03-25T23:50:26.521 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:50:26.522 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528353 2024-03-25T23:50:26.522 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:50:26.523 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-25T23:50:26.524 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528353 2024-03-25T23:50:26.524 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:50:26.525 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528353 2024-03-25T23:50:26.525 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528353' 2024-03-25T23:50:26.526 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.8 seq 193273528353 2024-03-25T23:50:26.526 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-25T23:50:26.849 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528353 -lt 193273528353 2024-03-25T23:50:26.849 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:50:26.850 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-365072220162 2024-03-25T23:50:26.851 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:50:26.851 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-25T23:50:26.852 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-365072220162 2024-03-25T23:50:26.852 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:50:26.854 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=365072220162 2024-03-25T23:50:26.854 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 365072220162' 2024-03-25T23:50:26.854 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.9 seq 365072220162 2024-03-25T23:50:26.854 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-25T23:50:27.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 365072220162 -lt 365072220162 2024-03-25T23:50:27.195 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:50:27.195 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:50:27.196 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:50:27.594 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:50:27.594 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:50:27.598 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:50:27.598 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:50:27.598 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:50:27.598 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:50:27.598 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:50:27.599 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:50:27.916 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:50:27.917 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:50:27.917 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:50:27.917 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:50:28.302 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:50:28.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:50:28.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:50:28.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-25T23:50:28.354 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-25T23:50:28.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2024-03-25T23:50:28.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-03-25T23:50:28.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-03-25T23:50:28.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=5 2024-03-25T23:50:28.360 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=9 2024-03-25T23:50:28.360 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-03-25T23:50:28.360 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-25T23:50:28.360 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:50:28.360 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:50:28.360 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 218863"' 2024-03-25T23:50:28.360 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 218863' 2024-03-25T23:50:28.361 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-03-25T23:50:28.361 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:50:28.361 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:50:28.361 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 218864"' 2024-03-25T23:50:28.361 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 218864' 2024-03-25T23:50:28.361 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-03-25T23:50:28.362 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 218863 218864' 2024-03-25T23:50:28.362 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-25T23:50:28.362 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-25T23:50:28.362 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-25T23:50:28.362 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 218863 2024-03-25T23:50:28.362 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/218866: /' 2024-03-25T23:50:28.362 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-03-25T23:50:28.363 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/218868: /' 2024-03-25T23:50:32.789 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:50:32.789 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:50:32.789 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-25T23:50:32.789 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:50:32.790 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-03-25T23:50:32.790 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:50:32.790 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:50:32.790 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-25T23:50:32.790 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:50:32.790 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-25T23:50:32.790 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:50:32.790 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:50:32.791 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:50:32.791 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:50:32.791 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:50:32.791 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:50:32.791 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-03-25T23:50:32.791 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:50:32.791 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:50:32.791 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-25T23:50:32.791 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:50:32.791 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-25T23:50:32.791 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-03-25T23:50:32.792 INFO:tasks.workunit.client.0.smithi129.stderr:218866: remove 1#2:eb822e21:::SOMETHING:head# 2024-03-25T23:50:32.792 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-25T23:50:32.792 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:50:32.792 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:50:32.792 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-25T23:50:32.792 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:50:32.792 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-25T23:50:32.792 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:50:32.792 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:50:32.795 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:50:32.795 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:50:32.795 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-25T23:50:32.795 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-25T23:50:32.795 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:50:32.795 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:50:32.795 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:50:32.795 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:50:32.795 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:50:32.795 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:50:32.795 INFO:tasks.workunit.client.0.smithi129.stderr:218866: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:50:32.795 INFO:tasks.workunit.client.0.smithi129.stderr:218866: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:50:32.796 INFO:tasks.workunit.client.0.smithi129.stderr:218866: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:50:32.796 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:50:32.796 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:50:32.796 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:50:32.796 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:50:32.796 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:50:32.796 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:50:32.796 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:50:32.796 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:50:32.796 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:50:32.796 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:50:32.796 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:50:32.796 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-25T23:50:32.796 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-25T23:50:32.797 INFO:tasks.workunit.client.0.smithi129.stderr:218866: start osd.5 2024-03-25T23:50:32.797 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:50:32.859 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:50:32.859 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:50:32.859 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-03-25T23:50:32.859 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:50:32.859 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2024-03-25T23:50:32.859 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:50:32.860 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:50:32.860 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-03-25T23:50:32.860 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:50:32.860 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-03-25T23:50:32.860 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:50:32.860 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:50:32.860 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:50:32.860 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:50:32.860 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:50:32.860 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:50:32.860 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2024-03-25T23:50:32.860 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:50:32.861 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:50:32.861 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-03-25T23:50:32.861 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:50:32.861 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-03-25T23:50:32.861 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2024-03-25T23:50:32.861 INFO:tasks.workunit.client.0.smithi129.stderr:218868: remove 2#2:eb822e21:::SOMETHING:head# 2024-03-25T23:50:32.861 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-03-25T23:50:32.861 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:50:32.861 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:50:32.861 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-03-25T23:50:32.861 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:50:32.861 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-03-25T23:50:32.861 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:50:32.861 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:50:32.862 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:50:32.863 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:50:32.863 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-25T23:50:32.863 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-25T23:50:32.863 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:50:32.863 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:50:32.863 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:50:32.863 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:50:32.863 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:50:32.863 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:50:32.863 INFO:tasks.workunit.client.0.smithi129.stderr:218868: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:50:32.863 INFO:tasks.workunit.client.0.smithi129.stderr:218868: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:50:32.863 INFO:tasks.workunit.client.0.smithi129.stderr:218868: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:50:32.863 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:50:32.863 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:50:32.864 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:50:32.864 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:50:32.864 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:50:32.864 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:50:32.864 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:50:32.864 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:50:32.864 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:50:32.864 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:50:32.864 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:50:32.864 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-03-25T23:50:32.864 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-03-25T23:50:32.864 INFO:tasks.workunit.client.0.smithi129.stderr:218868: start osd.9 2024-03-25T23:50:32.864 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:50:37.564 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-25T23:50:37.564 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-25T23:50:37.564 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:50:37.564 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:50:37.565 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:50:37.565 INFO:tasks.workunit.client.0.smithi129.stderr:218866: 2024-03-25T23:50:32.828+0000 7fc79b1d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:50:37.565 INFO:tasks.workunit.client.0.smithi129.stderr:218866: 2024-03-25T23:50:32.832+0000 7fc79b1d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:50:37.565 INFO:tasks.workunit.client.0.smithi129.stderr:218866: 2024-03-25T23:50:32.832+0000 7fc79b1d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:50:37.565 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-25T23:50:37.565 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:50:37.565 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-25T23:50:37.565 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:50:37.565 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:50:37.565 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:50:37.565 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:50:37.565 INFO:tasks.workunit.client.0.smithi129.stderr:218866: 0 2024-03-25T23:50:37.565 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:50:37.565 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:50:37.566 INFO:tasks.workunit.client.0.smithi129.stderr:218866: 2024-03-25T23:50:33.376+0000 7fc79b1d67c0 -1 Falling back to public interface 2024-03-25T23:50:37.566 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:50:37.566 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:50:37.566 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:50:37.566 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:50:37.566 INFO:tasks.workunit.client.0.smithi129.stderr:218866: 1 2024-03-25T23:50:37.566 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:50:37.566 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:50:37.566 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:50:37.566 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:50:37.566 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:50:37.566 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:50:37.566 INFO:tasks.workunit.client.0.smithi129.stderr:218866: 2 2024-03-25T23:50:37.567 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:50:37.567 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:50:37.567 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:50:37.567 INFO:tasks.workunit.client.0.smithi129.stderr:218866: 2024-03-25T23:50:36.492+0000 7fc79b1d67c0 -1 osd.5 86 log_to_monitors true 2024-03-25T23:50:37.567 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:50:37.567 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:50:37.567 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:50:37.567 INFO:tasks.workunit.client.0.smithi129.stderr:218866: 3 2024-03-25T23:50:37.567 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:50:37.567 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:50:37.571 INFO:tasks.workunit.client.0.smithi129.stderr:218866: osd.5 up in weight 1 up_from 93 up_thru 79 down_at 87 last_218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-03-25T23:50:37.572 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-03-25T23:50:37.572 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:50:37.572 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:50:37.572 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:50:37.572 INFO:tasks.workunit.client.0.smithi129.stderr:218868: 2024-03-25T23:50:32.892+0000 7f212b9ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:50:37.572 INFO:tasks.workunit.client.0.smithi129.stderr:218868: 2024-03-25T23:50:32.892+0000 7f212b9ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:50:37.572 INFO:tasks.workunit.client.0.smithi129.stderr:218868: 2024-03-25T23:50:32.896+0000 7f212b9ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:50:37.572 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-03-25T23:50:37.572 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:50:37.572 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-25T23:50:37.573 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:50:37.573 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:50:37.573 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:50:37.573 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:50:37.573 INFO:tasks.workunit.client.0.smithi129.stderr:218868: 0 2024-03-25T23:50:37.573 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:50:37.573 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:50:37.573 INFO:tasks.workunit.client.0.smithi129.stderr:218868: 2024-03-25T23:50:33.448+0000 7f212b9ca7c0 -1 Falling back to public interface 2024-03-25T23:50:37.573 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:50:37.573 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:50:37.573 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:50:37.574 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:50:37.574 INFO:tasks.workunit.client.0.smithi129.stderr:218868: 1 2024-03-25T23:50:37.574 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:50:37.574 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:50:37.574 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:50:37.574 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:50:37.574 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:50:37.574 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:50:37.574 INFO:tasks.workunit.client.0.smithi129.stderr:218868: 2 2024-03-25T23:50:37.574 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:50:37.574 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:50:37.575 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:50:37.575 INFO:tasks.workunit.client.0.smithi129.stderr:218868: 2024-03-25T23:50:36.656+0000 7f212b9ca7c0 -1 osd.9 87 log_to_monitors true 2024-03-25T23:50:37.575 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:50:37.575 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:50:37.575 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:50:37.575 INFO:tasks.workunit.client.0.smithi129.stderr:218868: 3 2024-03-25T23:50:37.575 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:50:37.575 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:50:38.593 INFO:tasks.workunit.client.0.smithi129.stderr:218868: osd.9 up in weight 1 up_from 93 up_thru 0 down_at 88 last_cclean_interval [79,86) [v2:127.0.0.1:6842/837470806,v1:127.0.0.1:6843/837470806] [v2:127.0.0.1:6844/837470806,v1:127.0.0.1:6845/837470806] exists,up 35c1c964-6ad5-4a39-91cc-74a5625271e2 2024-03-25T23:50:38.593 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:50:38.593 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:50:38.593 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:50:38.594 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:50:38.594 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:50:38.594 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:50:38.594 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:50:38.594 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:50:38.594 INFO:tasks.workunit.client.0.smithi129.stderr:218866: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:50:38.594 INFO:tasks.workunit.client.0.smithi129.stderr:218866: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:50:38.594 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:50:38.594 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:50:38.594 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:50:38.594 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:50:38.594 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:50:38.595 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:50:38.595 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:50:38.595 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:50:38.595 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:50:38.595 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:50:38.595 INFO:tasks.workunit.client.0.smithi129.stderr:218866: 1 2024-03-25T23:50:38.595 INFO:tasks.workunit.client.0.smithi129.stderr:218866: 2 2024-03-25T23:50:38.595 INFO:tasks.workunit.client.0.smithi129.stderr:218866: 3 2024-03-25T23:50:38.595 INFO:tasks.workunit.client.0.smithi129.stderr:218866: 4 2024-03-25T23:50:38.595 INFO:tasks.workunit.client.0.smithi129.stderr:218866: 5 2024-03-25T23:50:38.595 INFO:tasks.workunit.client.0.smithi129.stderr:218866: 6 2024-03-25T23:50:38.595 INFO:tasks.workunit.client.0.smithi129.stderr:218866: 7 2024-03-25T23:50:38.595 INFO:tasks.workunit.client.0.smithi129.stderr:218866: 8 2024-03-25T23:50:38.595 INFO:tasks.workunit.client.0.smithi129.stderr:218866: 9' 2024-03-25T23:50:38.596 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:50:38.596 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:50:38.596 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:50:38.596 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836581 2024-03-25T23:50:38.596 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836581 2024-03-25T23:50:38.596 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581' 2024-03-25T23:50:38.596 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:50:38.596 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:50:38.596 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673055 2024-03-25T23:50:38.596 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673055 2024-03-25T23:50:38.596 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673055' 2024-03-25T23:50:38.596 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:50:38.596 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/stlean_interval [85,87) [v2:127.0.0.1:6874/2805625846,v1:127.0.0.1:6875/2805625846] [v2:127.0.0.1:6876/2805625846,v1:127.0.0.1:6877/2805625846] exists,up e9f65c34-00ed-4379-b041-f45b4e8cc538 2024-03-25T23:50:38.596 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:50:38.597 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:50:38.597 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:50:38.597 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:50:38.597 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:50:38.597 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:50:38.597 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:50:38.597 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:50:38.597 INFO:tasks.workunit.client.0.smithi129.stderr:218868: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:50:38.597 INFO:tasks.workunit.client.0.smithi129.stderr:218868: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:50:38.597 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:50:38.597 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:50:38.597 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:50:38.597 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:50:38.598 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:50:38.598 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:50:38.598 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:50:38.598 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:50:38.598 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:50:38.598 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:50:38.598 INFO:tasks.workunit.client.0.smithi129.stderr:218868: 1 2024-03-25T23:50:38.598 INFO:tasks.workunit.client.0.smithi129.stderr:218868: 2 2024-03-25T23:50:38.598 INFO:tasks.workunit.client.0.smithi129.stderr:218868: 3 2024-03-25T23:50:38.598 INFO:tasks.workunit.client.0.smithi129.stderr:218868: 4 2024-03-25T23:50:38.598 INFO:tasks.workunit.client.0.smithi129.stderr:218868: 5 2024-03-25T23:50:38.598 INFO:tasks.workunit.client.0.smithi129.stderr:218868: 6 2024-03-25T23:50:38.598 INFO:tasks.workunit.client.0.smithi129.stderr:218868: 7 2024-03-25T23:50:38.598 INFO:tasks.workunit.client.0.smithi129.stderr:218868: 8 2024-03-25T23:50:38.599 INFO:tasks.workunit.client.0.smithi129.stderr:218868: 9' 2024-03-25T23:50:38.599 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:50:38.599 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:50:38.599 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:50:38.599 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836582 2024-03-25T23:50:38.599 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836582 2024-03-25T23:50:38.599 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582' 2024-03-25T23:50:38.599 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:50:38.599 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:50:38.599 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673056 2024-03-25T23:50:38.599 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673056 2024-03-25T23:50:38.600 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673056' 2024-03-25T23:50:38.600 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:50:40.159 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qaandalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:50:40.159 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509521 2024-03-25T23:50:40.159 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509521 2024-03-25T23:50:40.160 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673055 2-64424509521' 2024-03-25T23:50:40.160 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:50:40.160 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:50:40.160 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=296352743444 2024-03-25T23:50:40.160 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 296352743444 2024-03-25T23:50:40.160 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673055 2-64424509521 3-296352743444' 2024-03-25T23:50:40.160 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:50:40.160 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T23:50:40.160 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182470 2024-03-25T23:50:40.160 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182470 2024-03-25T23:50:40.160 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673055 2-64424509521 3-296352743444 4-107374182470' 2024-03-25T23:50:40.160 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:50:40.160 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T23:50:40.161 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=399431958530 2024-03-25T23:50:40.161 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 399431958530 2024-03-25T23:50:40.161 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673055 2-64424509521 3-296352743444 4-107374182470 5-399431958530' 2024-03-25T23:50:40.161 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:50:40.161 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T23:50:40.161 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855412 2024-03-25T23:50:40.161 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855412 2024-03-25T23:50:40.161 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673055 2-64424509521 3-296352743444 4-107374182470 5-399431958530 6-150323855412' 2024-03-25T23:50:40.161 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:50:40.161 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-25T23:50:40.161 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691884 2024-03-25T23:50:40.161 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691884 2024-03-25T23:50:40.161 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673055 2-64424509521 3-296352743444 4-107374182470 5-399431958530 6-150323855412 7-171798691884' 2024-03-25T23:50:40.161 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:50:40.162 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-25T23:50:40.163 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.cl/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:50:40.164 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509520 2024-03-25T23:50:40.164 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509520 2024-03-25T23:50:40.164 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673056 2-64424509520' 2024-03-25T23:50:40.164 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:50:40.164 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:50:40.164 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=296352743443 2024-03-25T23:50:40.164 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 296352743443 2024-03-25T23:50:40.164 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673056 2-64424509520 3-296352743443' 2024-03-25T23:50:40.164 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:50:40.164 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T23:50:40.164 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182471 2024-03-25T23:50:40.164 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182471 2024-03-25T23:50:40.164 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673056 2-64424509520 3-296352743443 4-107374182471' 2024-03-25T23:50:40.165 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:50:40.165 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T23:50:40.165 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=399431958531 2024-03-25T23:50:40.165 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 399431958531 2024-03-25T23:50:40.165 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673056 2-64424509520 3-296352743443 4-107374182471 5-399431958531' 2024-03-25T23:50:40.165 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:50:40.165 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T23:50:40.165 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855411 2024-03-25T23:50:40.165 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855411 2024-03-25T23:50:40.165 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673056 2-64424509520 3-296352743443 4-107374182471 5-399431958531 6-150323855411' 2024-03-25T23:50:40.165 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:50:40.165 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-25T23:50:40.165 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691885 2024-03-25T23:50:40.165 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691885 2024-03-25T23:50:40.166 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673056 2-64424509520 3-296352743443 4-107374182471 5-399431958531 6-150323855411 7-171798691885' 2024-03-25T23:50:40.166 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:50:40.166 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-25T23:50:40.918 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528358 2024-03-25T23:50:40.919 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528358 2024-03-25T23:50:40.919 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673056 2-64424509520 3-296352743443 4-107374182471 5-399431958531 6-150323855411 7-171798691885 8-193273528358' 2024-03-25T23:50:40.919 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:50:40.919 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-25T23:50:40.919 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=399431958531 2024-03-25T23:50:40.919 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 399431958531 2024-03-25T23:50:40.919 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673056 2-64424509520 3-296352743443 4-107374182471 5-399431958531 6-150323855411 7-171798691885 8-193273528358 9-399431958531' 2024-03-25T23:50:40.919 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:50:40.919 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836582 2024-03-25T23:50:40.920 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:50:40.920 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:50:40.920 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836582 2024-03-25T23:50:40.920 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:50:40.920 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836582 2024-03-25T23:50:40.920 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836582' 2024-03-25T23:50:40.920 INFO:tasks.workunit.client.0.smithi129.stderr:218868: waiting osd.0 seq 21474836582 2024-03-25T23:50:40.920 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:50:40.920 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836582 -lt 21474836582 2024-03-25T23:50:40.920 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:50:40.920 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673056 2024-03-25T23:50:40.920 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:50:40.921 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:50:40.921 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673056 2024-03-25T23:50:40.921 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:50:40.921 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673056 2024-03-25T23:50:40.921 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673056' 2024-03-25T23:50:40.921 INFO:tasks.workunit.client.0.smithi129.stderr:218868: waiting osd.1 seq 42949673056 2024-03-25T23:50:40.921 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:50:40.921 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673057 -lt 42949673056 2024-03-25T23:50:40.921 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:50:40.921 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509520 2024-03-25T23:50:40.921 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:50:40.922 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:50:40.922 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.cient.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528357 2024-03-25T23:50:40.922 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528357 2024-03-25T23:50:40.922 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673055 2-64424509521 3-296352743444 4-107374182470 5-399431958530 6-150323855412 7-171798691884 8-193273528357' 2024-03-25T23:50:40.922 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:50:40.922 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-25T23:50:40.923 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=399431958530 2024-03-25T23:50:40.923 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 399431958530 2024-03-25T23:50:40.923 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673055 2-64424509521 3-296352743444 4-107374182470 5-399431958530 6-150323855412 7-171798691884 8-193273528357 9-399431958530' 2024-03-25T23:50:40.923 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:50:40.923 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836581 2024-03-25T23:50:40.923 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:50:40.923 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:50:40.923 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836581 2024-03-25T23:50:40.923 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:50:40.923 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836581 2024-03-25T23:50:40.923 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836581' 2024-03-25T23:50:40.924 INFO:tasks.workunit.client.0.smithi129.stderr:218866: waiting osd.0 seq 21474836581 2024-03-25T23:50:40.924 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:50:40.924 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836582 -lt 21474836581 2024-03-25T23:50:40.924 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:50:40.924 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673055 2024-03-25T23:50:40.924 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:50:40.924 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:50:40.924 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673055 2024-03-25T23:50:40.924 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:50:40.924 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673055 2024-03-25T23:50:40.924 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673055' 2024-03-25T23:50:40.925 INFO:tasks.workunit.client.0.smithi129.stderr:218866: waiting osd.1 seq 42949673055 2024-03-25T23:50:40.925 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:50:40.925 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673057 -lt 42949673055 2024-03-25T23:50:40.925 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:50:40.925 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509521 2024-03-25T23:50:40.925 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:50:40.925 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:50:41.928 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.clielient.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509520 2024-03-25T23:50:41.928 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:50:41.928 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509520 2024-03-25T23:50:41.928 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509520' 2024-03-25T23:50:41.929 INFO:tasks.workunit.client.0.smithi129.stderr:218868: waiting osd.2 seq 64424509520 2024-03-25T23:50:41.929 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:50:41.929 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509521 -lt 64424509520 2024-03-25T23:50:41.929 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:50:41.929 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-296352743443 2024-03-25T23:50:41.929 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:50:41.929 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:50:41.929 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-296352743443 2024-03-25T23:50:41.929 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:50:41.929 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=296352743443 2024-03-25T23:50:41.929 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 296352743443' 2024-03-25T23:50:41.929 INFO:tasks.workunit.client.0.smithi129.stderr:218868: waiting osd.3 seq 296352743443 2024-03-25T23:50:41.930 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:50:41.930 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 296352743445 -lt 296352743443 2024-03-25T23:50:41.930 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:50:41.930 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182471 2024-03-25T23:50:41.930 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:50:41.930 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T23:50:41.930 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182471 2024-03-25T23:50:41.930 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:50:41.930 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182471 2024-03-25T23:50:41.930 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182471' 2024-03-25T23:50:41.930 INFO:tasks.workunit.client.0.smithi129.stderr:218868: waiting osd.4 seq 107374182471 2024-03-25T23:50:41.931 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T23:50:41.931 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182472 -lt 107374182471 2024-03-25T23:50:41.931 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:50:41.931 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-399431958531 2024-03-25T23:50:41.931 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:50:41.931 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T23:50:41.931 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-399431958531 2024-03-25T23:50:41.931 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:50:41.931 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=399431958531 2024-03-25T23:50:41.940 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waitint.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509521 2024-03-25T23:50:41.940 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:50:41.940 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509521 2024-03-25T23:50:41.940 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509521' 2024-03-25T23:50:41.940 INFO:tasks.workunit.client.0.smithi129.stderr:218866: waiting osd.2 seq 64424509521 2024-03-25T23:50:41.940 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:50:41.940 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509521 -lt 64424509521 2024-03-25T23:50:41.940 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:50:41.940 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-296352743444 2024-03-25T23:50:41.940 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:50:41.941 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:50:41.941 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-296352743444 2024-03-25T23:50:41.941 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:50:41.941 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=296352743444 2024-03-25T23:50:41.941 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 296352743444' 2024-03-25T23:50:41.941 INFO:tasks.workunit.client.0.smithi129.stderr:218866: waiting osd.3 seq 296352743444 2024-03-25T23:50:41.941 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:50:41.941 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 296352743445 -lt 296352743444 2024-03-25T23:50:41.941 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:50:41.941 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182470 2024-03-25T23:50:41.941 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:50:41.941 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T23:50:41.941 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182470 2024-03-25T23:50:41.941 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:50:41.942 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182470 2024-03-25T23:50:41.942 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182470' 2024-03-25T23:50:41.942 INFO:tasks.workunit.client.0.smithi129.stderr:218866: waiting osd.4 seq 107374182470 2024-03-25T23:50:41.942 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T23:50:41.942 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182472 -lt 107374182470 2024-03-25T23:50:41.942 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:50:41.942 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-399431958530 2024-03-25T23:50:41.942 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:50:41.942 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T23:50:41.942 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-399431958530 2024-03-25T23:50:41.942 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:50:41.942 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=399431958530 2024-03-25T23:50:43.220 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 399431958530' 2024-03-25T23:50:43.221 INFO:tasks.workunit.client.0.smithi129.stderr:218866: waiting osd.5 seq 399431958530 2024-03-25T23:50:43.221 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T23:50:43.221 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958531 -lt 399431958530 2024-03-25T23:50:43.221 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:50:43.221 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855412 2024-03-25T23:50:43.221 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:50:43.221 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-25T23:50:43.221 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855412 2024-03-25T23:50:43.221 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:50:43.221 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855412 2024-03-25T23:50:43.221 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855412' 2024-03-25T23:50:43.221 INFO:tasks.workunit.client.0.smithi129.stderr:218866: waiting osd.6 seq 150323855412 2024-03-25T23:50:43.222 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-25T23:50:43.222 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855413 -lt 150323855412 2024-03-25T23:50:43.222 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:50:43.222 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691884 2024-03-25T23:50:43.222 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:50:43.222 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-25T23:50:43.222 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691884 2024-03-25T23:50:43.222 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:50:43.222 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691884 2024-03-25T23:50:43.222 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691884' 2024-03-25T23:50:43.222 INFO:tasks.workunit.client.0.smithi129.stderr:218866: waiting osd.7 seq 171798691884 2024-03-25T23:50:43.222 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-25T23:50:43.222 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691886 -lt 171798691884 2024-03-25T23:50:43.222 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:50:43.223 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528357 2024-03-25T23:50:43.223 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:50:43.223 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-25T23:50:43.223 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528357 2024-03-25T23:50:43.223 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:50:43.223 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528357 2024-03-25T23:50:43.223 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528357' 2024-03-25T23:50:43.223 INFO:tasks.workunit.client.0.smithi129.stderr:218866: waiting osd.8 seq 193273528357 2024-03-25T23:50:43.223 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-25T23:50:43.223 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528359 -lt 193273528357 2024-03-25T23:50:43.225 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stng osd.5 seq 399431958531' 2024-03-25T23:50:43.225 INFO:tasks.workunit.client.0.smithi129.stderr:218868: waiting osd.5 seq 399431958531 2024-03-25T23:50:43.225 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T23:50:43.225 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958531 -lt 399431958531 2024-03-25T23:50:43.225 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:50:43.225 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855411 2024-03-25T23:50:43.225 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:50:43.225 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-25T23:50:43.225 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855411 2024-03-25T23:50:43.225 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:50:43.225 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855411 2024-03-25T23:50:43.225 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855411' 2024-03-25T23:50:43.225 INFO:tasks.workunit.client.0.smithi129.stderr:218868: waiting osd.6 seq 150323855411 2024-03-25T23:50:43.226 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-25T23:50:43.226 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855413 -lt 150323855411 2024-03-25T23:50:43.226 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:50:43.226 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691885 2024-03-25T23:50:43.226 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:50:43.226 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-25T23:50:43.226 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691885 2024-03-25T23:50:43.226 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:50:43.226 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691885 2024-03-25T23:50:43.226 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691885' 2024-03-25T23:50:43.226 INFO:tasks.workunit.client.0.smithi129.stderr:218868: waiting osd.7 seq 171798691885 2024-03-25T23:50:43.226 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-25T23:50:43.226 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691886 -lt 171798691885 2024-03-25T23:50:43.227 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:50:43.227 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528358 2024-03-25T23:50:43.227 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:50:43.227 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-25T23:50:43.227 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528358 2024-03-25T23:50:43.227 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:50:43.227 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528358 2024-03-25T23:50:43.227 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528358' 2024-03-25T23:50:43.227 INFO:tasks.workunit.client.0.smithi129.stderr:218868: waiting osd.8 seq 193273528358 2024-03-25T23:50:43.227 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-25T23:50:43.227 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528359 -lt 193273528358 2024-03-25T23:50:44.613 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pgats: for s in $seqs 2024-03-25T23:50:44.613 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-399431958530 2024-03-25T23:50:44.613 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:50:44.613 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-25T23:50:44.613 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-399431958530 2024-03-25T23:50:44.613 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:50:44.613 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=399431958530 2024-03-25T23:50:44.614 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 399431958530' 2024-03-25T23:50:44.614 INFO:tasks.workunit.client.0.smithi129.stderr:218866: waiting osd.9 seq 399431958530 2024-03-25T23:50:44.614 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-25T23:50:44.614 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958532 -lt 399431958530 2024-03-25T23:50:44.614 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:50:44.614 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:50:44.614 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:50:44.614 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:50:44.614 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:50:44.614 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:50:44.614 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:50:44.614 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:50:44.614 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:50:44.614 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:50:44.615 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:50:44.615 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:50:44.615 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:50:44.615 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:50:44.615 INFO:tasks.workunit.client.0.smithi129.stderr:218866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:50:44.615 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:50:44.615 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:50:44.615 INFO:tasks.workunit.client.0.smithi129.stderr:218866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:50:44.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:50:44.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-25T23:50:44.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 218864 2024-03-25T23:50:44.629 INFO:tasks.workunit.client.0.smithi129.stderr:_stats: for s in $seqs 2024-03-25T23:50:44.629 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-399431958531 2024-03-25T23:50:44.629 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:50:44.629 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-25T23:50:44.629 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-399431958531 2024-03-25T23:50:44.629 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:50:44.630 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=399431958531 2024-03-25T23:50:44.630 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 399431958531' 2024-03-25T23:50:44.630 INFO:tasks.workunit.client.0.smithi129.stderr:218868: waiting osd.9 seq 399431958531 2024-03-25T23:50:44.630 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-25T23:50:44.630 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958532 -lt 399431958531 2024-03-25T23:50:44.630 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:50:44.630 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:50:44.630 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:50:44.630 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:50:44.630 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:50:44.630 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:50:44.630 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:50:44.630 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:50:44.631 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:50:44.631 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:50:44.631 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:50:44.631 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:50:44.631 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:50:44.631 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:50:44.631 INFO:tasks.workunit.client.0.smithi129.stderr:218868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:50:44.631 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:50:44.631 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:50:44.631 INFO:tasks.workunit.client.0.smithi129.stderr:218868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:50:44.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:50:44.632 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-25T23:50:44.632 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-25T23:50:44.632 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-25T23:50:44.632 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-03-25T23:50:44.632 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-25T23:50:44.632 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-03-25T23:50:44.632 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-25T23:50:44.632 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-25T23:50:44.632 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-25T23:50:44.632 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-25T23:50:44.943 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-03-25T23:50:44.943 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-03-25T23:50:44.943 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-25T23:50:44.944 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-25T23:50:44.944 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:50:44.944 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:50:44.944 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:50:44.944 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:50:45.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-25T23:50:11.921237+0000 2024-03-25T23:50:45.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-25T23:50:45.553 INFO:tasks.workunit.client.0.smithi129.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-25T23:50:45.569 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-25T23:50:11.921237+0000 2024-03-25T23:50:45.570 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-25T23:50:45.570 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-25T23:50:11.921237+0000 2024-03-25T23:50:45.570 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-25T23:50:45.570 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-25T23:50:45.570 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:50:45.570 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:50:45.570 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:50:45.570 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:50:45.570 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:50:45.570 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:50:45.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:50:11.921237+0000 '>' 2024-03-25T23:50:11.921237+0000 2024-03-25T23:50:45.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:50:46.875 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:50:46.875 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:50:46.876 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:50:46.876 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:50:46.877 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:50:46.877 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:50:46.877 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:50:47.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:50:11.921237+0000 '>' 2024-03-25T23:50:11.921237+0000 2024-03-25T23:50:47.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:50:48.196 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:50:48.196 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:50:48.197 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:50:48.197 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:50:48.197 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:50:48.197 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:50:48.198 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:50:48.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:50:11.921237+0000 '>' 2024-03-25T23:50:11.921237+0000 2024-03-25T23:50:48.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:50:49.519 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:50:49.519 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:50:49.520 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:50:49.520 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:50:49.520 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:50:49.521 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:50:49.521 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:50:49.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:50:11.921237+0000 '>' 2024-03-25T23:50:11.921237+0000 2024-03-25T23:50:49.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:50:50.841 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:50:50.841 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:50:50.842 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:50:50.842 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:50:50.842 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:50:50.843 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:50:50.843 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:50:51.163 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:50:11.921237+0000 '>' 2024-03-25T23:50:11.921237+0000 2024-03-25T23:50:51.164 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:50:52.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:50:52.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:50:52.166 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:50:52.166 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:50:52.166 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:50:52.167 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:50:52.167 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:50:52.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:50:46.304500+0000 '>' 2024-03-25T23:50:11.921237+0000 2024-03-25T23:50:52.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-25T23:50:52.490 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-03-25T23:50:52.490 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-25T23:50:52.490 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:50:52.490 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:50:52.490 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 222296"' 2024-03-25T23:50:52.490 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 222296' 2024-03-25T23:50:52.490 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-25T23:50:52.490 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:50:52.490 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:50:52.490 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 222297"' 2024-03-25T23:50:52.490 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 222297' 2024-03-25T23:50:52.490 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-03-25T23:50:52.490 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 222296 222297' 2024-03-25T23:50:52.491 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-25T23:50:52.491 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-25T23:50:52.491 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 222296 2024-03-25T23:50:52.491 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-25T23:50:52.491 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-25T23:50:52.491 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/222300: /' 2024-03-25T23:50:52.491 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/222301: /' 2024-03-25T23:50:53.464 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:50:53.464 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:50:53.464 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-03-25T23:50:53.464 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:50:53.464 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-25T23:50:53.464 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:50:53.464 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:50:53.464 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-03-25T23:50:53.464 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:50:53.465 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-03-25T23:50:53.465 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:50:53.465 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:50:53.465 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:50:53.465 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:50:53.465 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:50:53.465 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:50:53.465 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-25T23:50:53.465 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:50:53.465 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:50:53.465 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-03-25T23:50:53.465 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:50:53.465 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-03-25T23:50:53.465 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-03-25T23:50:53.466 INFO:tasks.workunit.client.0.smithi129.stderr:222301: _ 2024-03-25T23:50:53.466 INFO:tasks.workunit.client.0.smithi129.stderr:222301: hinfo_key 2024-03-25T23:50:53.466 INFO:tasks.workunit.client.0.smithi129.stderr:222301: snapset 2024-03-25T23:50:53.466 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-03-25T23:50:53.466 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:50:53.466 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:50:53.466 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-03-25T23:50:53.466 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:50:53.466 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-03-25T23:50:53.466 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:50:53.466 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:50:53.467 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helper222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:50:53.467 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:50:53.467 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-25T23:50:53.467 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:50:53.467 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-25T23:50:53.467 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:50:53.467 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:50:53.467 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-25T23:50:53.467 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:50:53.467 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-25T23:50:53.468 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:50:53.468 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:50:53.468 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:50:53.468 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:50:53.468 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:50:53.468 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:50:53.468 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-25T23:50:53.468 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:50:53.468 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:50:53.468 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-25T23:50:53.469 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:50:53.469 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-25T23:50:53.469 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-03-25T23:50:53.469 INFO:tasks.workunit.client.0.smithi129.stderr:222300: _ 2024-03-25T23:50:53.469 INFO:tasks.workunit.client.0.smithi129.stderr:222300: hinfo_key 2024-03-25T23:50:53.469 INFO:tasks.workunit.client.0.smithi129.stderr:222300: snapset 2024-03-25T23:50:53.469 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-25T23:50:53.469 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:50:53.469 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:50:53.469 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-25T23:50:53.469 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:50:53.469 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-25T23:50:53.469 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:50:53.470 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:50:53.470 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:50:53.470 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:50:53.470 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-25T23:50:53.470 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-25T23:50:53.470 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:50:53.471 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:50:53.471 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:50:53.471 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:50:53.471 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:50:53.471 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:50:53.471 INFO:tasks.workunit.client.0.smithi129.stderr:222301: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:50:53.471 INFO:tasks.workunit.client.0.smithi129.stderr:222301: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:50:53.471 INFO:tasks.workunit.client.0.smithi129.stderr:222301: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:50:53.471 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:50:53.471 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:50:53.471 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:50:53.471 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:50:53.471 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:50:53.471 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:50:53.472 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:50:53.472 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:50:53.472 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:50:53.472 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:50:53.472 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:50:53.472 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-03-25T23:50:53.472 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-03-25T23:50:53.472 INFO:tasks.workunit.client.0.smithi129.stderr:222301: start osd.9 2024-03-25T23:50:53.472 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recs.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:50:53.472 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:50:53.472 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-25T23:50:53.472 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-25T23:50:53.472 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:50:53.472 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:50:53.473 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:50:53.473 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:50:53.473 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:50:53.473 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:50:53.473 INFO:tasks.workunit.client.0.smithi129.stderr:222300: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:50:53.473 INFO:tasks.workunit.client.0.smithi129.stderr:222300: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:50:53.473 INFO:tasks.workunit.client.0.smithi129.stderr:222300: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:50:53.473 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:50:53.473 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:50:53.473 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:50:53.473 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:50:53.473 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:50:53.473 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:50:53.473 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:50:53.474 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:50:53.474 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:50:53.474 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:50:53.474 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:50:53.474 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-25T23:50:53.474 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-25T23:50:53.474 INFO:tasks.workunit.client.0.smithi129.stderr:222300: start osd.5 2024-03-25T23:50:58.397 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:50:58.397 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-25T23:50:58.397 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-25T23:50:58.397 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:50:58.397 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:50:58.397 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:50:58.398 INFO:tasks.workunit.client.0.smithi129.stderr:222300: 2024-03-25T23:50:53.499+0000 7f7623c317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:50:58.398 INFO:tasks.workunit.client.0.smithi129.stderr:222300: 2024-03-25T23:50:53.503+0000 7f7623c317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:50:58.398 INFO:tasks.workunit.client.0.smithi129.stderr:222300: 2024-03-25T23:50:53.507+0000 7f7623c317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:50:58.398 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-25T23:50:58.398 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:50:58.398 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-25T23:50:58.398 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:50:58.398 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:50:58.398 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:50:58.398 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:50:58.398 INFO:tasks.workunit.client.0.smithi129.stderr:222300: 0 2024-03-25T23:50:58.399 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:50:58.399 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:50:58.399 INFO:tasks.workunit.client.0.smithi129.stderr:222300: 2024-03-25T23:50:54.079+0000 7f7623c317c0 -1 Falling back to public interface 2024-03-25T23:50:58.399 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:50:58.399 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:50:58.399 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:50:58.399 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:50:58.399 INFO:tasks.workunit.client.0.smithi129.stderr:222300: 1 2024-03-25T23:50:58.399 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:50:58.399 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:50:58.399 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:50:58.400 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:50:58.400 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:50:58.400 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:50:58.400 INFO:tasks.workunit.client.0.smithi129.stderr:222300: 2 2024-03-25T23:50:58.400 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:50:58.400 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:50:58.400 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:50:58.400 INFO:tasks.workunit.client.0.smithi129.stderr:222300: 2024-03-25T23:50:57.735+0000 7f7623c317c0 -1 osd.5 95 log_to_monitors true 2024-03-25T23:50:58.400 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:50:58.400 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:50:58.400 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:50:58.400 INFO:tasks.workunit.client.0.smithi129.stderr:222300: 3 2024-03-25T23:50:58.401 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:50:58.401 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:50:58.544 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/covery_ops 2024-03-25T23:50:58.544 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-03-25T23:50:58.544 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-03-25T23:50:58.545 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:50:58.545 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:50:58.545 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:50:58.545 INFO:tasks.workunit.client.0.smithi129.stderr:222301: 2024-03-25T23:50:53.503+0000 7f0b5b7857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:50:58.545 INFO:tasks.workunit.client.0.smithi129.stderr:222301: 2024-03-25T23:50:53.515+0000 7f0b5b7857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:50:58.545 INFO:tasks.workunit.client.0.smithi129.stderr:222301: 2024-03-25T23:50:53.515+0000 7f0b5b7857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:50:58.545 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-03-25T23:50:58.545 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:50:58.545 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-25T23:50:58.545 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:50:58.545 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:50:58.546 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:50:58.546 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:50:58.546 INFO:tasks.workunit.client.0.smithi129.stderr:222301: 0 2024-03-25T23:50:58.546 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:50:58.546 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:50:58.546 INFO:tasks.workunit.client.0.smithi129.stderr:222301: 2024-03-25T23:50:54.107+0000 7f0b5b7857c0 -1 Falling back to public interface 2024-03-25T23:50:58.546 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:50:58.546 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:50:58.546 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:50:58.546 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:50:58.546 INFO:tasks.workunit.client.0.smithi129.stderr:222301: 1 2024-03-25T23:50:58.546 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:50:58.547 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:50:58.547 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:50:58.547 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:50:58.547 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:50:58.547 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:50:58.547 INFO:tasks.workunit.client.0.smithi129.stderr:222301: 2 2024-03-25T23:50:58.547 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:50:58.547 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:50:58.547 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:50:58.547 INFO:tasks.workunit.client.0.smithi129.stderr:222301: 2024-03-25T23:50:57.835+0000 7f0b5b7857c0 -1 osd.9 95 log_to_monitors true 2024-03-25T23:50:58.547 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:50:58.547 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:50:58.548 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:50:58.548 INFO:tasks.workunit.client.0.smithi129.stderr:222301: 3 2024-03-25T23:50:58.548 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:50:58.548 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:51:01.138 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:51:01.138 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:51:01.138 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:51:01.138 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:51:01.138 INFO:tasks.workunit.client.0.smithi129.stderr:222300: 4 2024-03-25T23:51:01.138 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:51:01.139 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:51:01.139 INFO:tasks.workunit.client.0.smithi129.stderr:222300: osd.5 up in weight 1 up_from 100 up_thru 100 down_at 96 last_clean_interval [93,95) [v2:127.0.0.1:6842/4193835252,v1:127.0.0.1:6843/4193835252] [v2:127.0.0.1:6844/4193835252,v1:127.0.0.1:6845/4193835252] exists,up 35c1c964-6ad5-4a39-91cc-74a5625271e2 2024-03-25T23:51:01.139 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:51:01.139 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:51:01.139 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:51:01.139 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:51:01.139 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:51:01.139 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:51:01.139 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:51:01.139 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:51:01.139 INFO:tasks.workunit.client.0.smithi129.stderr:222300: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:51:01.139 INFO:tasks.workunit.client.0.smithi129.stderr:222300: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:51:01.139 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:51:01.140 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:51:01.140 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:51:01.140 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:51:01.140 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:51:01.140 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:51:01.140 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:51:01.140 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:51:01.140 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:51:01.140 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:51:01.140 INFO:tasks.workunit.client.0.smithi129.stderr:222300: 1 2024-03-25T23:51:01.140 INFO:tasks.workunit.client.0.smithi129.stderr:222300: 2 2024-03-25T23:51:01.140 INFO:tasks.workunit.client.0.smithi129.stderr:222300: 3 2024-03-25T23:51:01.140 INFO:tasks.workunit.client.0.smithi129.stderr:222300: 4 2024-03-25T23:51:01.140 INFO:tasks.workunit.client.0.smithi129.stderr:222300: 5 2024-03-25T23:51:01.141 INFO:tasks.workunit.client.0.smithi129.stderr:222300: 6 2024-03-25T23:51:01.141 INFO:tasks.workunit.client.0.smithi129.stderr:222300: 7 2024-03-25T23:51:01.141 INFO:tasks.workunit.client.0.smithi129.stderr:222300: 8 2024-03-25T23:51:01.141 INFO:tasks.workunit.client.0.smithi129.stderr:222300: 9' 2024-03-25T23:51:01.141 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:51:01.141 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:01.141 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:51:01.141 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836589 2024-03-25T23:51:01.141 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836589 2024-03-25T23:51:01.141 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589' 2024-03-25T23:51:01.141 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $eph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:51:01.141 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:51:01.141 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:51:01.141 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:51:01.142 INFO:tasks.workunit.client.0.smithi129.stderr:222301: 4 2024-03-25T23:51:01.142 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:51:01.142 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:51:01.142 INFO:tasks.workunit.client.0.smithi129.stderr:222301: osd.9 up in weight 1 up_from 101 up_thru 0 down_at 96 last_clean_interval [93,95) [v2:127.0.0.1:6874/4054064032,v1:127.0.0.1:6875/4054064032] [v2:127.0.0.1:6876/4054064032,v1:127.0.0.1:6877/4054064032] exists,up e9f65c34-00ed-4379-b041-f45b4e8cc538 2024-03-25T23:51:01.142 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:51:01.142 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:51:01.142 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:51:01.142 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:51:01.142 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:51:01.142 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:51:01.142 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:51:01.142 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:51:01.142 INFO:tasks.workunit.client.0.smithi129.stderr:222301: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:51:01.143 INFO:tasks.workunit.client.0.smithi129.stderr:222301: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:51:01.143 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:51:01.143 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:51:01.143 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:51:01.143 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:51:01.143 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:51:01.143 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:51:01.143 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:51:01.143 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:51:01.143 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:51:01.143 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:51:01.143 INFO:tasks.workunit.client.0.smithi129.stderr:222301: 1 2024-03-25T23:51:01.143 INFO:tasks.workunit.client.0.smithi129.stderr:222301: 2 2024-03-25T23:51:01.143 INFO:tasks.workunit.client.0.smithi129.stderr:222301: 3 2024-03-25T23:51:01.143 INFO:tasks.workunit.client.0.smithi129.stderr:222301: 4 2024-03-25T23:51:01.144 INFO:tasks.workunit.client.0.smithi129.stderr:222301: 5 2024-03-25T23:51:01.144 INFO:tasks.workunit.client.0.smithi129.stderr:222301: 6 2024-03-25T23:51:01.144 INFO:tasks.workunit.client.0.smithi129.stderr:222301: 7 2024-03-25T23:51:01.144 INFO:tasks.workunit.client.0.smithi129.stderr:222301: 8 2024-03-25T23:51:01.144 INFO:tasks.workunit.client.0.smithi129.stderr:222301: 9' 2024-03-25T23:51:01.144 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:51:01.144 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:01.144 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:51:01.144 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836588 2024-03-25T23:51:01.144 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836588 2024-03-25T23:51:01.144 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836588' 2024-03-25T23:51:02.358 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:02.358 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:51:02.358 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673062 2024-03-25T23:51:02.358 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673062 2024-03-25T23:51:02.358 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836588 1-42949673062' 2024-03-25T23:51:02.358 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:02.358 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:51:02.359 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509527 2024-03-25T23:51:02.359 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509527 2024-03-25T23:51:02.359 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836588 1-42949673062 2-64424509527' 2024-03-25T23:51:02.359 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:02.359 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:51:02.359 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=296352743451 2024-03-25T23:51:02.359 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 296352743451 2024-03-25T23:51:02.359 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836588 1-42949673062 2-64424509527 3-296352743451' 2024-03-25T23:51:02.359 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:02.359 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T23:51:02.359 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182478 2024-03-25T23:51:02.359 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182478 2024-03-25T23:51:02.359 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836588 1-42949673062 2-64424509527 3-296352743451 4-107374182478' 2024-03-25T23:51:02.359 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:02.360 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T23:51:02.360 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=429496729603 2024-03-25T23:51:02.360 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 429496729603 2024-03-25T23:51:02.360 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836588 1-42949673062 2-64424509527 3-296352743451 4-107374182478 5-429496729603' 2024-03-25T23:51:02.360 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:02.360 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T23:51:02.360 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855419 2024-03-25T23:51:02.360 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855419 2024-03-25T23:51:02.360 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836588 1-42949673062 2-64424509527 3-296352743451 4-107374182478 5-429496729603 6-150323855419' 2024-03-25T23:51:02.360 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:02.360 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-25T23:51:02.361 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2ids 2024-03-25T23:51:02.361 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:51:02.361 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673063 2024-03-25T23:51:02.361 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673063 2024-03-25T23:51:02.361 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673063' 2024-03-25T23:51:02.361 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:02.361 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:51:02.361 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509528 2024-03-25T23:51:02.361 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509528 2024-03-25T23:51:02.361 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673063 2-64424509528' 2024-03-25T23:51:02.361 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:02.361 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:51:02.362 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=296352743450 2024-03-25T23:51:02.362 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 296352743450 2024-03-25T23:51:02.362 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673063 2-64424509528 3-296352743450' 2024-03-25T23:51:02.362 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:02.362 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T23:51:02.362 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182477 2024-03-25T23:51:02.362 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182477 2024-03-25T23:51:02.362 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673063 2-64424509528 3-296352743450 4-107374182477' 2024-03-25T23:51:02.362 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:02.362 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T23:51:02.362 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=429496729602 2024-03-25T23:51:02.362 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 429496729602 2024-03-25T23:51:02.362 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673063 2-64424509528 3-296352743450 4-107374182477 5-429496729602' 2024-03-25T23:51:02.362 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:02.363 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T23:51:02.363 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855418 2024-03-25T23:51:02.363 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855418 2024-03-25T23:51:02.363 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673063 2-64424509528 3-296352743450 4-107374182477 5-429496729602 6-150323855418' 2024-03-25T23:51:02.363 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:02.363 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-25T23:51:02.932 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691892 2024-03-25T23:51:02.932 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691892 2024-03-25T23:51:02.932 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673063 2-64424509528 3-296352743450 4-107374182477 5-429496729602 6-150323855418 7-171798691892' 2024-03-25T23:51:02.932 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:02.932 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-25T23:51:02.932 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528364 2024-03-25T23:51:02.932 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528364 2024-03-25T23:51:02.932 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673063 2-64424509528 3-296352743450 4-107374182477 5-429496729602 6-150323855418 7-171798691892 8-193273528364' 2024-03-25T23:51:02.932 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:02.932 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-25T23:51:02.932 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=433791696898 2024-03-25T23:51:02.932 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 433791696898 2024-03-25T23:51:02.933 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673063 2-64424509528 3-296352743450 4-107374182477 5-429496729602 6-150323855418 7-171798691892 8-193273528364 9-433791696898' 2024-03-25T23:51:02.933 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:02.933 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836589 2024-03-25T23:51:02.933 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:02.933 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:51:02.933 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836589 2024-03-25T23:51:02.933 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:02.933 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836589 2024-03-25T23:51:02.933 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836589' 2024-03-25T23:51:02.933 INFO:tasks.workunit.client.0.smithi129.stderr:222300: waiting osd.0 seq 21474836589 2024-03-25T23:51:02.933 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:51:02.933 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836589 -lt 21474836589 2024-03-25T23:51:02.933 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:02.933 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673063 2024-03-25T23:51:02.934 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:02.934 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:51:02.934 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673063 2024-03-25T23:51:02.934 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:02.934 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673063 2024-03-25T23:51:02.934 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673063' 2024-03-25T23:51:02.934 INFO:tasks.workunit.client.0.smithi129.stderr:222300: waiting osd.1 seq 42949673063 2024-03-25T23:51:02.934 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:221202: flush_pg_stats: seq=171798691891 2024-03-25T23:51:02.935 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691891 2024-03-25T23:51:02.935 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836588 1-42949673062 2-64424509527 3-296352743451 4-107374182478 5-429496729603 6-150323855419 7-171798691891' 2024-03-25T23:51:02.935 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:02.935 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-25T23:51:02.935 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528365 2024-03-25T23:51:02.935 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528365 2024-03-25T23:51:02.935 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836588 1-42949673062 2-64424509527 3-296352743451 4-107374182478 5-429496729603 6-150323855419 7-171798691891 8-193273528365' 2024-03-25T23:51:02.935 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:02.935 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-25T23:51:02.935 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=433791696899 2024-03-25T23:51:02.935 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 433791696899 2024-03-25T23:51:02.935 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836588 1-42949673062 2-64424509527 3-296352743451 4-107374182478 5-429496729603 6-150323855419 7-171798691891 8-193273528365 9-433791696899' 2024-03-25T23:51:02.935 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:02.936 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836588 2024-03-25T23:51:02.936 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:02.936 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:51:02.936 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836588 2024-03-25T23:51:02.936 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:02.936 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836588 2024-03-25T23:51:02.936 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836588' 2024-03-25T23:51:02.936 INFO:tasks.workunit.client.0.smithi129.stderr:222301: waiting osd.0 seq 21474836588 2024-03-25T23:51:02.936 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:51:02.936 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836589 -lt 21474836588 2024-03-25T23:51:02.936 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:02.936 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673062 2024-03-25T23:51:02.936 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:02.937 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:51:02.937 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673062 2024-03-25T23:51:02.937 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:02.937 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673062 2024-03-25T23:51:02.937 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673062' 2024-03-25T23:51:02.937 INFO:tasks.workunit.client.0.smithi129.stderr:222301: waiting osd.1 seq 42949673062 2024-03-25T23:51:04.270 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214:4: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:51:04.270 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673063 -lt 42949673063 2024-03-25T23:51:04.270 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:04.270 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509528 2024-03-25T23:51:04.270 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:04.270 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:51:04.270 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509528 2024-03-25T23:51:04.270 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:04.270 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509528 2024-03-25T23:51:04.270 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509528' 2024-03-25T23:51:04.271 INFO:tasks.workunit.client.0.smithi129.stderr:222300: waiting osd.2 seq 64424509528 2024-03-25T23:51:04.271 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:51:04.271 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509528 -lt 64424509528 2024-03-25T23:51:04.271 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:04.271 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-296352743450 2024-03-25T23:51:04.271 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:04.271 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:51:04.271 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-296352743450 2024-03-25T23:51:04.271 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:04.271 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=296352743450 2024-03-25T23:51:04.271 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 296352743450' 2024-03-25T23:51:04.271 INFO:tasks.workunit.client.0.smithi129.stderr:222300: waiting osd.3 seq 296352743450 2024-03-25T23:51:04.271 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:51:04.271 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 296352743451 -lt 296352743450 2024-03-25T23:51:04.271 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:04.272 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182477 2024-03-25T23:51:04.272 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:04.272 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T23:51:04.272 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182477 2024-03-25T23:51:04.272 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:04.272 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182477 2024-03-25T23:51:04.272 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182477' 2024-03-25T23:51:04.272 INFO:tasks.workunit.client.0.smithi129.stderr:222300: waiting osd.4 seq 107374182477 2024-03-25T23:51:04.272 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T23:51:04.272 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182478 -lt 107374182477 2024-03-25T23:51:04.272 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:04.272 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:04.274 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ub flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:51:04.274 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673063 -lt 42949673062 2024-03-25T23:51:04.274 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:04.275 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509527 2024-03-25T23:51:04.275 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:04.275 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:51:04.275 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509527 2024-03-25T23:51:04.275 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:04.275 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509527 2024-03-25T23:51:04.275 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509527' 2024-03-25T23:51:04.275 INFO:tasks.workunit.client.0.smithi129.stderr:222301: waiting osd.2 seq 64424509527 2024-03-25T23:51:04.275 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:51:04.275 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509528 -lt 64424509527 2024-03-25T23:51:04.275 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:04.275 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-296352743451 2024-03-25T23:51:04.275 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:04.275 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:51:04.276 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-296352743451 2024-03-25T23:51:04.276 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:04.276 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=296352743451 2024-03-25T23:51:04.276 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 296352743451' 2024-03-25T23:51:04.276 INFO:tasks.workunit.client.0.smithi129.stderr:222301: waiting osd.3 seq 296352743451 2024-03-25T23:51:04.276 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:51:04.276 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 296352743451 -lt 296352743451 2024-03-25T23:51:04.276 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:04.276 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182478 2024-03-25T23:51:04.276 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:04.276 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T23:51:04.276 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182478 2024-03-25T23:51:04.276 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:04.276 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182478 2024-03-25T23:51:04.277 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182478' 2024-03-25T23:51:04.277 INFO:tasks.workunit.client.0.smithi129.stderr:222301: waiting osd.4 seq 107374182478 2024-03-25T23:51:04.277 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T23:51:04.277 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182478 -lt 107374182478 2024-03-25T23:51:04.277 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:04.277 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-429496729603 2024-03-25T23:51:05.220 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:05.220 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T23:51:05.220 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-429496729603 2024-03-25T23:51:05.220 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:05.221 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=429496729603 2024-03-25T23:51:05.221 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 429496729603' 2024-03-25T23:51:05.221 INFO:tasks.workunit.client.0.smithi129.stderr:222301: waiting osd.5 seq 429496729603 2024-03-25T23:51:05.221 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T23:51:05.221 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 429496729604 -lt 429496729603 2024-03-25T23:51:05.221 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:05.221 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855419 2024-03-25T23:51:05.221 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:05.221 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-25T23:51:05.221 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855419 2024-03-25T23:51:05.221 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:05.221 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855419 2024-03-25T23:51:05.221 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855419' 2024-03-25T23:51:05.221 INFO:tasks.workunit.client.0.smithi129.stderr:222301: waiting osd.6 seq 150323855419 2024-03-25T23:51:05.222 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-25T23:51:05.222 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855419 -lt 150323855419 2024-03-25T23:51:05.222 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:05.222 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691891 2024-03-25T23:51:05.222 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:05.222 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-25T23:51:05.222 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691891 2024-03-25T23:51:05.222 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:05.222 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691891 2024-03-25T23:51:05.222 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691891' 2024-03-25T23:51:05.222 INFO:tasks.workunit.client.0.smithi129.stderr:222301: waiting osd.7 seq 171798691891 2024-03-25T23:51:05.222 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-25T23:51:05.223 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691892 -lt 171798691891 2024-03-25T23:51:05.223 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:05.223 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528365 2024-03-25T23:51:05.223 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:05.223 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-25T23:51:05.223 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528365 2024-03-25T23:51:05.233 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_puntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-429496729602 2024-03-25T23:51:05.233 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T23:51:05.233 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-429496729602 2024-03-25T23:51:05.233 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:05.234 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=429496729602 2024-03-25T23:51:05.234 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 429496729602' 2024-03-25T23:51:05.234 INFO:tasks.workunit.client.0.smithi129.stderr:222300: waiting osd.5 seq 429496729602 2024-03-25T23:51:05.234 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T23:51:05.234 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 429496729604 -lt 429496729602 2024-03-25T23:51:05.234 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:05.234 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855418 2024-03-25T23:51:05.234 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:05.234 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-25T23:51:05.234 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855418 2024-03-25T23:51:05.234 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:05.234 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855418 2024-03-25T23:51:05.234 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855418' 2024-03-25T23:51:05.235 INFO:tasks.workunit.client.0.smithi129.stderr:222300: waiting osd.6 seq 150323855418 2024-03-25T23:51:05.235 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-25T23:51:05.235 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855419 -lt 150323855418 2024-03-25T23:51:05.235 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:05.235 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691892 2024-03-25T23:51:05.235 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:05.235 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-25T23:51:05.235 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691892 2024-03-25T23:51:05.235 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:05.235 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691892 2024-03-25T23:51:05.235 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691892' 2024-03-25T23:51:05.235 INFO:tasks.workunit.client.0.smithi129.stderr:222300: waiting osd.7 seq 171798691892 2024-03-25T23:51:05.235 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-25T23:51:05.235 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691892 -lt 171798691892 2024-03-25T23:51:05.235 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:05.236 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528364 2024-03-25T23:51:05.236 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:05.236 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-25T23:51:05.236 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528364 2024-03-25T23:51:06.923 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flushg_stats: cut -d - -f 2 2024-03-25T23:51:06.923 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528365 2024-03-25T23:51:06.923 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528365' 2024-03-25T23:51:06.924 INFO:tasks.workunit.client.0.smithi129.stderr:222301: waiting osd.8 seq 193273528365 2024-03-25T23:51:06.924 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-25T23:51:06.924 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528365 -lt 193273528365 2024-03-25T23:51:06.924 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:06.924 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-433791696899 2024-03-25T23:51:06.924 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:06.924 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-25T23:51:06.924 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-433791696899 2024-03-25T23:51:06.924 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:06.925 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=433791696899 2024-03-25T23:51:06.925 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 433791696899' 2024-03-25T23:51:06.925 INFO:tasks.workunit.client.0.smithi129.stderr:222301: waiting osd.9 seq 433791696899 2024-03-25T23:51:06.925 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-25T23:51:06.925 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696900 -lt 433791696899 2024-03-25T23:51:06.925 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:51:06.925 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:51:06.925 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:51:06.926 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:51:06.926 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:51:06.926 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:51:06.926 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:51:06.926 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:51:06.926 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:51:06.926 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:51:06.926 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:51:06.926 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T23:51:06.927 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:51:06.927 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:51:06.927 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:51:06.927 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-25T23:51:06.927 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-25T23:51:06.927 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-25T23:51:06.945 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/u_pg_stats: cut -d - -f 2 2024-03-25T23:51:06.945 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528364 2024-03-25T23:51:06.945 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528364' 2024-03-25T23:51:06.945 INFO:tasks.workunit.client.0.smithi129.stderr:222300: waiting osd.8 seq 193273528364 2024-03-25T23:51:06.946 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-25T23:51:06.946 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528365 -lt 193273528364 2024-03-25T23:51:06.946 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:06.946 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-433791696898 2024-03-25T23:51:06.946 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:06.946 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-25T23:51:06.946 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-433791696898 2024-03-25T23:51:06.946 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:06.946 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=433791696898 2024-03-25T23:51:06.946 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 433791696898' 2024-03-25T23:51:06.946 INFO:tasks.workunit.client.0.smithi129.stderr:222300: waiting osd.9 seq 433791696898 2024-03-25T23:51:06.946 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-25T23:51:06.946 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696900 -lt 433791696898 2024-03-25T23:51:06.947 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:51:06.947 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:51:06.947 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:51:06.947 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:51:06.947 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:51:06.947 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:51:06.947 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:51:06.947 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:51:06.947 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:51:06.947 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:51:06.947 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:51:06.947 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T23:51:06.947 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:51:06.947 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:51:06.947 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:51:06.948 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-25T23:51:06.948 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-25T23:51:06.948 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-25T23:51:08.290 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /homebuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-25T23:51:08.290 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:51:08.290 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-25T23:51:08.290 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:51:08.290 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:51:08.290 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:51:08.290 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:51:08.290 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:51:08.290 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:51:08.290 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:51:08.291 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:51:08.291 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T23:51:08.291 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:51:08.291 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:51:08.291 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:51:08.291 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-25T23:51:08.291 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-25T23:51:08.291 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-25T23:51:08.291 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-25T23:51:08.291 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-25T23:51:08.291 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-25T23:51:08.291 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-25T23:51:08.291 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-25T23:51:08.291 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-25T23:51:08.292 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-25T23:51:08.292 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-25T23:51:08.292 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-25T23:51:08.292 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:51:08.292 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-25T23:51:08.292 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:51:08.292 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:51:08.292 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-25T23:51:08.292 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:51:08.292 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-25T23:51:08.292 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:51:08.292 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:51:08.292 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:51:08.292 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:51:08.293 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:51:08.293 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:51:08.293 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:51:08.293 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:51:08.293 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T23:51:08.293 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:51:08.293 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:51:08.293 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:51:08.293 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-25T23:51:08.293 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-25T23:51:08.293 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-25T23:51:08.293 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-25T23:51:08.293 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-25T23:51:08.293 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-25T23:51:08.293 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-25T23:51:08.294 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-25T23:51:08.294 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-25T23:51:08.294 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-25T23:51:08.294 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-25T23:51:08.294 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-25T23:51:08.294 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:51:08.294 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-25T23:51:08.294 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:51:08.294 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:51:08.936 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clo.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:51:08.936 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:51:08.937 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:51:08.937 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:51:08.937 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:51:08.937 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:51:08.937 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:51:08.937 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:51:08.937 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:51:08.937 INFO:tasks.workunit.client.0.smithi129.stderr:222301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:51:08.937 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:51:08.937 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:51:08.937 INFO:tasks.workunit.client.0.smithi129.stderr:222301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:51:08.937 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:51:08.955 INFO:tasks.workunit.client.0.smithi129.stderr:ne.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:51:08.955 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:51:08.955 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:51:08.955 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:51:08.955 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:51:08.955 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:51:08.955 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:51:08.955 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:51:08.955 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:51:08.955 INFO:tasks.workunit.client.0.smithi129.stderr:222300: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:51:08.955 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:51:08.955 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:51:08.955 INFO:tasks.workunit.client.0.smithi129.stderr:222300: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:51:08.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:51:08.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-25T23:51:08.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 222297 2024-03-25T23:51:08.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-25T23:51:08.956 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-25T23:51:08.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-25T23:51:08.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-03-25T23:51:08.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-25T23:51:08.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-25T23:51:09.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-25T23:51:09.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2024-03-25T23:51:09.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-03-25T23:51:09.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-03-25T23:51:09.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-03-25T23:51:09.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=5 2024-03-25T23:51:09.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-03-25T23:51:09.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-25T23:51:09.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:51:09.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:51:09.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 225458"' 2024-03-25T23:51:09.038 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 225458' 2024-03-25T23:51:09.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-25T23:51:09.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:51:09.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:51:09.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-25T23:51:09.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 225460"' 2024-03-25T23:51:09.038 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 225460' 2024-03-25T23:51:09.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-03-25T23:51:09.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 225458 225460' 2024-03-25T23:51:09.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-25T23:51:09.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-25T23:51:09.039 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 225458 2024-03-25T23:51:09.039 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-25T23:51:09.039 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/225461: /' 2024-03-25T23:51:09.039 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/225463: /' 2024-03-25T23:51:12.672 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:51:12.672 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:51:12.673 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-25T23:51:12.673 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:51:12.673 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-03-25T23:51:12.673 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:51:12.673 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:51:12.673 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-25T23:51:12.673 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:51:12.673 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-25T23:51:12.673 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:51:12.673 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:51:12.673 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:51:12.673 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:51:12.674 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:51:12.674 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:51:12.674 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-03-25T23:51:12.674 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:51:12.674 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:51:12.674 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-25T23:51:12.674 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:51:12.674 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-25T23:51:12.674 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-03-25T23:51:12.674 INFO:tasks.workunit.client.0.smithi129.stderr:225461: remove 0#2:eb822e21:::SOMETHING:head# 2024-03-25T23:51:12.674 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-25T23:51:12.674 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:51:12.674 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:51:12.674 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-25T23:51:12.675 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:51:12.675 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-25T23:51:12.675 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:51:12.675 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:51:12.678 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:51:12.678 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:51:12.678 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-25T23:51:12.678 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-25T23:51:12.678 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:51:12.678 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:51:12.678 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:51:12.678 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:51:12.678 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:51:12.678 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:51:12.678 INFO:tasks.workunit.client.0.smithi129.stderr:225461: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:51:12.679 INFO:tasks.workunit.client.0.smithi129.stderr:225461: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:51:12.679 INFO:tasks.workunit.client.0.smithi129.stderr:225461: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:51:12.679 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:51:12.679 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:51:12.679 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:51:12.679 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:51:12.679 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:51:12.679 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:51:12.679 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:51:12.679 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:51:12.679 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:51:12.679 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:51:12.679 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:51:12.679 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-25T23:51:12.680 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-25T23:51:12.680 INFO:tasks.workunit.client.0.smithi129.stderr:225461: start osd.3 2024-03-25T23:51:12.680 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:51:13.074 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:51:13.074 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:51:13.074 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-25T23:51:13.074 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:51:13.074 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-03-25T23:51:13.074 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:51:13.074 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:51:13.075 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-25T23:51:13.075 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:51:13.075 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-25T23:51:13.075 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:51:13.075 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:51:13.075 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:51:13.075 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:51:13.075 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:51:13.075 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:51:13.075 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-03-25T23:51:13.075 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:51:13.075 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:51:13.075 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-25T23:51:13.075 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:51:13.076 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-25T23:51:13.076 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-03-25T23:51:13.076 INFO:tasks.workunit.client.0.smithi129.stderr:225463: remove 1#2:eb822e21:::SOMETHING:head# 2024-03-25T23:51:13.076 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-25T23:51:13.076 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:51:13.076 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:51:13.076 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-25T23:51:13.076 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:51:13.076 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-25T23:51:13.076 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:51:13.076 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:51:13.078 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:51:13.078 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:51:13.078 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-25T23:51:13.078 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-25T23:51:13.078 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:51:13.078 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:51:13.078 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:51:13.078 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:51:13.078 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:51:13.078 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:51:13.078 INFO:tasks.workunit.client.0.smithi129.stderr:225463: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:51:13.078 INFO:tasks.workunit.client.0.smithi129.stderr:225463: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:51:13.079 INFO:tasks.workunit.client.0.smithi129.stderr:225463: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:51:13.079 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:51:13.079 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:51:13.079 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:51:13.079 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:51:13.079 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:51:13.079 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:51:13.079 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:51:13.079 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:51:13.079 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:51:13.079 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:51:13.079 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:51:13.079 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-25T23:51:13.079 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-25T23:51:13.079 INFO:tasks.workunit.client.0.smithi129.stderr:225463: start osd.5 2024-03-25T23:51:13.080 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:51:17.479 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-25T23:51:17.479 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-25T23:51:17.479 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:51:17.480 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:51:17.480 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:51:17.480 INFO:tasks.workunit.client.0.smithi129.stderr:225461: 2024-03-25T23:51:12.711+0000 7ff1d56407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:51:17.480 INFO:tasks.workunit.client.0.smithi129.stderr:225461: 2024-03-25T23:51:12.711+0000 7ff1d56407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:51:17.480 INFO:tasks.workunit.client.0.smithi129.stderr:225461: 2024-03-25T23:51:12.715+0000 7ff1d56407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:51:17.480 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-25T23:51:17.480 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:51:17.481 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-25T23:51:17.481 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:51:17.481 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:51:17.481 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:51:17.481 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:51:17.481 INFO:tasks.workunit.client.0.smithi129.stderr:225461: 0 2024-03-25T23:51:17.481 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:51:17.481 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:51:17.481 INFO:tasks.workunit.client.0.smithi129.stderr:225461: 2024-03-25T23:51:13.287+0000 7ff1d56407c0 -1 Falling back to public interface 2024-03-25T23:51:17.481 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:51:17.481 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:51:17.482 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:51:17.482 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:51:17.482 INFO:tasks.workunit.client.0.smithi129.stderr:225461: 1 2024-03-25T23:51:17.482 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:51:17.482 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:51:17.482 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:51:17.482 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:51:17.483 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:51:17.483 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:51:17.483 INFO:tasks.workunit.client.0.smithi129.stderr:225461: 2 2024-03-25T23:51:17.483 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:51:17.483 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:51:17.483 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:51:17.483 INFO:tasks.workunit.client.0.smithi129.stderr:225461: 2024-03-25T23:51:16.131+0000 7ff1d56407c0 -1 osd.3 103 log_to_monitors true 2024-03-25T23:51:17.483 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:51:17.484 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:51:17.484 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:51:17.484 INFO:tasks.workunit.client.0.smithi129.stderr:225461: 3 2024-03-25T23:51:17.484 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:51:17.484 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:51:17.905 INFO:tasks.workunit.client.0.smithi129.stderr:225461: osd.3 up in weight 1 up_from 108 up_thru 101 down_at 104 l225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-25T23:51:17.905 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-25T23:51:17.905 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:51:17.905 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:51:17.905 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:51:17.905 INFO:tasks.workunit.client.0.smithi129.stderr:225463: 2024-03-25T23:51:13.107+0000 7ffb987b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:51:17.905 INFO:tasks.workunit.client.0.smithi129.stderr:225463: 2024-03-25T23:51:13.127+0000 7ffb987b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:51:17.905 INFO:tasks.workunit.client.0.smithi129.stderr:225463: 2024-03-25T23:51:13.127+0000 7ffb987b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:51:17.905 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-25T23:51:17.905 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:51:17.906 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-25T23:51:17.906 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:51:17.906 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:51:17.906 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:51:17.906 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:51:17.906 INFO:tasks.workunit.client.0.smithi129.stderr:225463: 0 2024-03-25T23:51:17.906 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:51:17.906 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:51:17.906 INFO:tasks.workunit.client.0.smithi129.stderr:225463: 2024-03-25T23:51:13.679+0000 7ffb987b47c0 -1 Falling back to public interface 2024-03-25T23:51:17.906 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:51:17.906 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:51:17.906 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:51:17.906 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:51:17.906 INFO:tasks.workunit.client.0.smithi129.stderr:225463: 1 2024-03-25T23:51:17.907 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:51:17.907 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:51:17.907 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:51:17.907 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:51:17.907 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:51:17.907 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:51:17.907 INFO:tasks.workunit.client.0.smithi129.stderr:225463: 2 2024-03-25T23:51:17.907 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:51:17.907 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:51:17.907 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:51:17.907 INFO:tasks.workunit.client.0.smithi129.stderr:225463: 2024-03-25T23:51:16.767+0000 7ffb987b47c0 -1 osd.5 103 log_to_monitors true 2024-03-25T23:51:17.907 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:51:17.907 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:51:17.907 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:51:17.907 INFO:tasks.workunit.client.0.smithi129.stderr:225463: 3 2024-03-25T23:51:17.908 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:51:17.908 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:51:18.470 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpeast_clean_interval [69,103) [v2:127.0.0.1:6826/4082470295,v1:127.0.0.1:6827/4082470295] [v2:127.0.0.1:6828/4082470295,v1:127.0.0.1:6829/4082470295] exists,up 711eac53-c0bb-40a8-a592-40574bbc51ac 2024-03-25T23:51:18.470 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:51:18.470 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:51:18.470 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:51:18.471 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:51:18.471 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:51:18.471 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:51:18.471 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:51:18.471 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:51:18.471 INFO:tasks.workunit.client.0.smithi129.stderr:225461: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:51:18.471 INFO:tasks.workunit.client.0.smithi129.stderr:225461: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:51:18.471 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:51:18.471 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:51:18.471 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:51:18.471 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:51:18.471 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:51:18.471 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:51:18.471 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:51:18.471 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:51:18.472 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:51:18.472 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:51:18.472 INFO:tasks.workunit.client.0.smithi129.stderr:225461: 1 2024-03-25T23:51:18.472 INFO:tasks.workunit.client.0.smithi129.stderr:225461: 2 2024-03-25T23:51:18.472 INFO:tasks.workunit.client.0.smithi129.stderr:225461: 3 2024-03-25T23:51:18.472 INFO:tasks.workunit.client.0.smithi129.stderr:225461: 4 2024-03-25T23:51:18.472 INFO:tasks.workunit.client.0.smithi129.stderr:225461: 5 2024-03-25T23:51:18.472 INFO:tasks.workunit.client.0.smithi129.stderr:225461: 6 2024-03-25T23:51:18.472 INFO:tasks.workunit.client.0.smithi129.stderr:225461: 7 2024-03-25T23:51:18.472 INFO:tasks.workunit.client.0.smithi129.stderr:225461: 8 2024-03-25T23:51:18.472 INFO:tasks.workunit.client.0.smithi129.stderr:225461: 9' 2024-03-25T23:51:18.472 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:51:18.472 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:18.472 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:51:18.472 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836593 2024-03-25T23:51:18.473 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836593 2024-03-25T23:51:18.473 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593' 2024-03-25T23:51:18.473 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:18.473 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:51:18.473 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673067 2024-03-25T23:51:18.473 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673067 2024-03-25T23:51:18.473 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673067' 2024-03-25T23:51:18.473 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:20.461 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.clienrs.sh:985: wait_for_osd: sleep 1 2024-03-25T23:51:20.461 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:51:20.461 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:51:20.461 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:51:20.462 INFO:tasks.workunit.client.0.smithi129.stderr:225463: 4 2024-03-25T23:51:20.462 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:51:20.462 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:51:20.462 INFO:tasks.workunit.client.0.smithi129.stderr:225463: osd.5 up in weight 1 up_from 111 up_thru 111 down_at 104 last_clean_interval [100,103) [v2:127.0.0.1:6842/3427386,v1:127.0.0.1:6843/3427386] [v2:127.0.0.1:6844/3427386,v1:127.0.0.1:6845/3427386] exists,up 35c1c964-6ad5-4a39-91cc-74a5625271e2 2024-03-25T23:51:20.462 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:51:20.462 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:51:20.462 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:51:20.463 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:51:20.463 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:51:20.463 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:51:20.463 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:51:20.463 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:51:20.463 INFO:tasks.workunit.client.0.smithi129.stderr:225463: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:51:20.463 INFO:tasks.workunit.client.0.smithi129.stderr:225463: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:51:20.463 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:51:20.463 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:51:20.463 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:51:20.463 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:51:20.463 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:51:20.463 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:51:20.463 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:51:20.464 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:51:20.464 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:51:20.464 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:51:20.464 INFO:tasks.workunit.client.0.smithi129.stderr:225463: 1 2024-03-25T23:51:20.464 INFO:tasks.workunit.client.0.smithi129.stderr:225463: 2 2024-03-25T23:51:20.464 INFO:tasks.workunit.client.0.smithi129.stderr:225463: 3 2024-03-25T23:51:20.464 INFO:tasks.workunit.client.0.smithi129.stderr:225463: 4 2024-03-25T23:51:20.464 INFO:tasks.workunit.client.0.smithi129.stderr:225463: 5 2024-03-25T23:51:20.464 INFO:tasks.workunit.client.0.smithi129.stderr:225463: 6 2024-03-25T23:51:20.464 INFO:tasks.workunit.client.0.smithi129.stderr:225463: 7 2024-03-25T23:51:20.464 INFO:tasks.workunit.client.0.smithi129.stderr:225463: 8 2024-03-25T23:51:20.464 INFO:tasks.workunit.client.0.smithi129.stderr:225463: 9' 2024-03-25T23:51:20.464 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:51:20.464 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:20.465 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:51:20.465 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836594 2024-03-25T23:51:20.465 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836594 2024-03-25T23:51:20.465 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836594' 2024-03-25T23:51:20.465 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:20.771 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //homet.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:51:20.772 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509532 2024-03-25T23:51:20.772 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509532 2024-03-25T23:51:20.772 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673067 2-64424509532' 2024-03-25T23:51:20.772 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:20.772 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:51:20.772 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=463856467970 2024-03-25T23:51:20.772 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 463856467970 2024-03-25T23:51:20.772 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673067 2-64424509532 3-463856467970' 2024-03-25T23:51:20.772 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:20.772 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T23:51:20.772 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182482 2024-03-25T23:51:20.772 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182482 2024-03-25T23:51:20.773 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673067 2-64424509532 3-463856467970 4-107374182482' 2024-03-25T23:51:20.773 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:20.773 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T23:51:20.773 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=476741369858 2024-03-25T23:51:20.773 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 476741369858 2024-03-25T23:51:20.773 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673067 2-64424509532 3-463856467970 4-107374182482 5-476741369858' 2024-03-25T23:51:20.773 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:20.773 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T23:51:20.773 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855423 2024-03-25T23:51:20.773 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855423 2024-03-25T23:51:20.773 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673067 2-64424509532 3-463856467970 4-107374182482 5-476741369858 6-150323855423' 2024-03-25T23:51:20.773 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:20.773 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-25T23:51:20.773 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691896 2024-03-25T23:51:20.773 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691896 2024-03-25T23:51:20.774 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673067 2-64424509532 3-463856467970 4-107374182482 5-476741369858 6-150323855423 7-171798691896' 2024-03-25T23:51:20.774 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:20.774 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-25T23:51:21.578 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528369 2024-03-25T23:51:21.578 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528369 2024-03-25T23:51:21.578 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673067 2-64424509532 3-463856467970 4-107374182482 5-476741369858 6-150323855423 7-171798691896 8-193273528369' 2024-03-25T23:51:21.579 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:21.579 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-25T23:51:21.579 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=433791696904 2024-03-25T23:51:21.579 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 433791696904 2024-03-25T23:51:21.579 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673067 2-64424509532 3-463856467970 4-107374182482 5-476741369858 6-150323855423 7-171798691896 8-193273528369 9-433791696904' 2024-03-25T23:51:21.579 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:21.579 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836593 2024-03-25T23:51:21.579 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:21.579 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:51:21.579 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836593 2024-03-25T23:51:21.579 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:21.579 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836593 2024-03-25T23:51:21.579 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836593' 2024-03-25T23:51:21.579 INFO:tasks.workunit.client.0.smithi129.stderr:225461: waiting osd.0 seq 21474836593 2024-03-25T23:51:21.580 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:51:21.580 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836593 -lt 21474836593 2024-03-25T23:51:21.580 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:21.580 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673067 2024-03-25T23:51:21.580 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:21.580 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:51:21.580 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673067 2024-03-25T23:51:21.580 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:21.580 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673067 2024-03-25T23:51:21.580 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673067' 2024-03-25T23:51:21.580 INFO:tasks.workunit.client.0.smithi129.stderr:225461: waiting osd.1 seq 42949673067 2024-03-25T23:51:21.580 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:51:21.580 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673069 -lt 42949673067 2024-03-25T23:51:21.580 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:21.581 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509532 2024-03-25T23:51:21.581 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:21.581 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:51:21.704 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/c/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:51:21.704 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673069 2024-03-25T23:51:21.704 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673069 2024-03-25T23:51:21.704 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836594 1-42949673069' 2024-03-25T23:51:21.705 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:21.705 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:51:21.705 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509534 2024-03-25T23:51:21.705 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509534 2024-03-25T23:51:21.705 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836594 1-42949673069 2-64424509534' 2024-03-25T23:51:21.705 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:21.705 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:51:21.705 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=463856467971 2024-03-25T23:51:21.705 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 463856467971 2024-03-25T23:51:21.705 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836594 1-42949673069 2-64424509534 3-463856467971' 2024-03-25T23:51:21.705 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:21.705 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T23:51:21.705 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182484 2024-03-25T23:51:21.705 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182484 2024-03-25T23:51:21.706 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836594 1-42949673069 2-64424509534 3-463856467971 4-107374182484' 2024-03-25T23:51:21.706 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:21.706 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T23:51:21.706 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=476741369859 2024-03-25T23:51:21.706 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 476741369859 2024-03-25T23:51:21.706 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836594 1-42949673069 2-64424509534 3-463856467971 4-107374182484 5-476741369859' 2024-03-25T23:51:21.706 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:21.706 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T23:51:21.706 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855425 2024-03-25T23:51:21.706 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855425 2024-03-25T23:51:21.706 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836594 1-42949673069 2-64424509534 3-463856467971 4-107374182484 5-476741369859 6-150323855425' 2024-03-25T23:51:21.706 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:21.706 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-25T23:51:22.285 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691898 2024-03-25T23:51:22.285 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691898 2024-03-25T23:51:22.286 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836594 1-42949673069 2-64424509534 3-463856467971 4-107374182484 5-476741369859 6-150323855425 7-171798691898' 2024-03-25T23:51:22.286 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:22.286 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-25T23:51:22.286 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528371 2024-03-25T23:51:22.286 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528371 2024-03-25T23:51:22.287 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836594 1-42949673069 2-64424509534 3-463856467971 4-107374182484 5-476741369859 6-150323855425 7-171798691898 8-193273528371' 2024-03-25T23:51:22.287 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:22.287 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-25T23:51:22.287 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=433791696905 2024-03-25T23:51:22.287 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 433791696905 2024-03-25T23:51:22.287 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836594 1-42949673069 2-64424509534 3-463856467971 4-107374182484 5-476741369859 6-150323855425 7-171798691898 8-193273528371 9-433791696905' 2024-03-25T23:51:22.287 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:22.288 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836594 2024-03-25T23:51:22.288 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:22.288 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:51:22.288 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836594 2024-03-25T23:51:22.288 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:22.288 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836594 2024-03-25T23:51:22.288 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836594' 2024-03-25T23:51:22.289 INFO:tasks.workunit.client.0.smithi129.stderr:225463: waiting osd.0 seq 21474836594 2024-03-25T23:51:22.289 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:51:22.289 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836595 -lt 21474836594 2024-03-25T23:51:22.289 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:22.289 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:22.289 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673069 2024-03-25T23:51:22.289 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:51:22.290 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673069 2024-03-25T23:51:22.290 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:22.290 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673069 2024-03-25T23:51:22.290 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673069' 2024-03-25T23:51:22.290 INFO:tasks.workunit.client.0.smithi129.stderr:225463: waiting osd.1 seq 42949673069 2024-03-25T23:51:22.568 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats:lone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509532 2024-03-25T23:51:22.568 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:22.568 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509532 2024-03-25T23:51:22.568 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509532' 2024-03-25T23:51:22.568 INFO:tasks.workunit.client.0.smithi129.stderr:225461: waiting osd.2 seq 64424509532 2024-03-25T23:51:22.568 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:51:22.568 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509534 -lt 64424509532 2024-03-25T23:51:22.568 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:22.568 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-463856467970 2024-03-25T23:51:22.568 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:22.568 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:51:22.568 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-463856467970 2024-03-25T23:51:22.569 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:22.569 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=463856467970 2024-03-25T23:51:22.569 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 463856467970' 2024-03-25T23:51:22.569 INFO:tasks.workunit.client.0.smithi129.stderr:225461: waiting osd.3 seq 463856467970 2024-03-25T23:51:22.569 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:51:22.569 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 463856467971 -lt 463856467970 2024-03-25T23:51:22.569 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:22.569 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182482 2024-03-25T23:51:22.569 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:22.569 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T23:51:22.569 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182482 2024-03-25T23:51:22.569 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:22.569 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182482 2024-03-25T23:51:22.570 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182482' 2024-03-25T23:51:22.570 INFO:tasks.workunit.client.0.smithi129.stderr:225461: waiting osd.4 seq 107374182482 2024-03-25T23:51:22.570 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T23:51:22.570 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182484 -lt 107374182482 2024-03-25T23:51:22.570 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:22.570 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-476741369858 2024-03-25T23:51:22.570 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:22.570 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T23:51:22.570 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-476741369858 2024-03-25T23:51:22.570 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:22.570 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=476741369858 2024-03-25T23:51:23.568 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo ceph osd last-stat-seq 1 2024-03-25T23:51:23.568 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673069 -lt 42949673069 2024-03-25T23:51:23.568 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:23.568 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509534 2024-03-25T23:51:23.568 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:23.568 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:51:23.568 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509534 2024-03-25T23:51:23.568 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:23.568 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509534 2024-03-25T23:51:23.568 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509534' 2024-03-25T23:51:23.568 INFO:tasks.workunit.client.0.smithi129.stderr:225463: waiting osd.2 seq 64424509534 2024-03-25T23:51:23.568 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:51:23.568 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509534 -lt 64424509534 2024-03-25T23:51:23.569 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:23.569 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-463856467971 2024-03-25T23:51:23.569 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:23.569 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:51:23.569 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-463856467971 2024-03-25T23:51:23.569 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:23.569 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=463856467971 2024-03-25T23:51:23.569 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 463856467971' 2024-03-25T23:51:23.569 INFO:tasks.workunit.client.0.smithi129.stderr:225463: waiting osd.3 seq 463856467971 2024-03-25T23:51:23.569 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:51:23.569 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 463856467971 -lt 463856467971 2024-03-25T23:51:23.569 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:23.569 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182484 2024-03-25T23:51:23.569 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:23.569 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T23:51:23.570 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182484 2024-03-25T23:51:23.570 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:23.570 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182484 2024-03-25T23:51:23.570 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182484' 2024-03-25T23:51:23.570 INFO:tasks.workunit.client.0.smithi129.stderr:225463: waiting osd.4 seq 107374182484 2024-03-25T23:51:23.570 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T23:51:23.570 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182484 -lt 107374182484 2024-03-25T23:51:23.570 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:23.570 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-476741369859 2024-03-25T23:51:23.893 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtes'waiting osd.5 seq 476741369858' 2024-03-25T23:51:23.893 INFO:tasks.workunit.client.0.smithi129.stderr:225461: waiting osd.5 seq 476741369858 2024-03-25T23:51:23.893 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T23:51:23.893 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 476741369858 -lt 476741369858 2024-03-25T23:51:23.893 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:23.893 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855423 2024-03-25T23:51:23.893 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:23.893 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-25T23:51:23.894 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855423 2024-03-25T23:51:23.894 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:23.894 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855423 2024-03-25T23:51:23.894 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855423' 2024-03-25T23:51:23.894 INFO:tasks.workunit.client.0.smithi129.stderr:225461: waiting osd.6 seq 150323855423 2024-03-25T23:51:23.894 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-25T23:51:23.894 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855424 -lt 150323855423 2024-03-25T23:51:23.894 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:23.894 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691896 2024-03-25T23:51:23.894 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:23.894 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-25T23:51:23.894 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691896 2024-03-25T23:51:23.894 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:23.895 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691896 2024-03-25T23:51:23.895 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691896' 2024-03-25T23:51:23.895 INFO:tasks.workunit.client.0.smithi129.stderr:225461: waiting osd.7 seq 171798691896 2024-03-25T23:51:23.895 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-25T23:51:23.895 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691898 -lt 171798691896 2024-03-25T23:51:23.895 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:23.895 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528369 2024-03-25T23:51:23.895 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:23.895 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-25T23:51:23.895 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528369 2024-03-25T23:51:23.895 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:23.895 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528369 2024-03-25T23:51:23.895 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528369' 2024-03-25T23:51:23.895 INFO:tasks.workunit.client.0.smithi129.stderr:225461: waiting osd.8 seq 193273528369 2024-03-25T23:51:23.896 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-25T23:51:23.896 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528371 -lt 193273528369 2024-03-25T23:51:24.529 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flt/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:24.529 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T23:51:24.529 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-476741369859 2024-03-25T23:51:24.529 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:24.529 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=476741369859 2024-03-25T23:51:24.529 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 476741369859' 2024-03-25T23:51:24.529 INFO:tasks.workunit.client.0.smithi129.stderr:225463: waiting osd.5 seq 476741369859 2024-03-25T23:51:24.529 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T23:51:24.529 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 476741369860 -lt 476741369859 2024-03-25T23:51:24.529 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:24.529 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855425 2024-03-25T23:51:24.530 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:24.530 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-25T23:51:24.530 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855425 2024-03-25T23:51:24.530 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:24.530 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855425 2024-03-25T23:51:24.530 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855425' 2024-03-25T23:51:24.530 INFO:tasks.workunit.client.0.smithi129.stderr:225463: waiting osd.6 seq 150323855425 2024-03-25T23:51:24.530 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-25T23:51:24.530 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855425 -lt 150323855425 2024-03-25T23:51:24.530 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:24.530 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691898 2024-03-25T23:51:24.530 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:24.530 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-25T23:51:24.531 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691898 2024-03-25T23:51:24.531 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:24.531 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691898 2024-03-25T23:51:24.531 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691898' 2024-03-25T23:51:24.531 INFO:tasks.workunit.client.0.smithi129.stderr:225463: waiting osd.7 seq 171798691898 2024-03-25T23:51:24.531 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-25T23:51:24.531 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691898 -lt 171798691898 2024-03-25T23:51:24.531 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:24.531 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528371 2024-03-25T23:51:24.531 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:24.531 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-25T23:51:24.531 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528371 2024-03-25T23:51:25.305 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -dush_pg_stats: for s in $seqs 2024-03-25T23:51:25.306 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-433791696904 2024-03-25T23:51:25.306 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:25.306 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-25T23:51:25.306 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-433791696904 2024-03-25T23:51:25.306 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:25.306 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=433791696904 2024-03-25T23:51:25.306 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 433791696904' 2024-03-25T23:51:25.306 INFO:tasks.workunit.client.0.smithi129.stderr:225461: waiting osd.9 seq 433791696904 2024-03-25T23:51:25.306 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-25T23:51:25.306 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696906 -lt 433791696904 2024-03-25T23:51:25.306 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:51:25.307 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:51:25.307 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:51:25.307 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:51:25.307 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:51:25.307 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:51:25.307 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:51:25.307 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:51:25.307 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:51:25.307 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:51:25.307 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:51:25.307 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:51:25.307 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:51:25.307 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:51:25.308 INFO:tasks.workunit.client.0.smithi129.stderr:225461: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:51:25.308 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:51:25.308 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:51:25.308 INFO:tasks.workunit.client.0.smithi129.stderr:225461: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:51:25.308 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:51:25.308 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-25T23:51:25.308 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 225460 2024-03-25T23:51:26.244 INFO:tasks.workunit.client.0.smithi129.stderr: - -f 2 2024-03-25T23:51:26.244 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528371 2024-03-25T23:51:26.244 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528371' 2024-03-25T23:51:26.244 INFO:tasks.workunit.client.0.smithi129.stderr:225463: waiting osd.8 seq 193273528371 2024-03-25T23:51:26.245 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-25T23:51:26.245 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528371 -lt 193273528371 2024-03-25T23:51:26.245 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:26.245 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-433791696905 2024-03-25T23:51:26.245 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:26.245 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-25T23:51:26.245 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-433791696905 2024-03-25T23:51:26.245 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:26.245 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=433791696905 2024-03-25T23:51:26.245 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 433791696905' 2024-03-25T23:51:26.245 INFO:tasks.workunit.client.0.smithi129.stderr:225463: waiting osd.9 seq 433791696905 2024-03-25T23:51:26.245 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-25T23:51:26.245 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696906 -lt 433791696905 2024-03-25T23:51:26.246 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:51:26.246 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:51:26.246 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:51:26.246 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:51:26.246 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:51:26.246 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:51:26.246 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:51:26.246 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:51:26.246 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:51:26.246 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:51:26.246 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:51:26.246 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:51:26.246 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:51:26.246 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:51:26.247 INFO:tasks.workunit.client.0.smithi129.stderr:225463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:51:26.247 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:51:26.247 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:51:26.247 INFO:tasks.workunit.client.0.smithi129.stderr:225463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:51:26.247 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:51:26.247 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-25T23:51:26.247 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-25T23:51:26.247 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-25T23:51:26.247 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-03-25T23:51:26.247 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-25T23:51:26.248 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-03-25T23:51:26.248 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-25T23:51:26.248 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-25T23:51:26.248 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-25T23:51:26.248 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-25T23:51:26.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-03-25T23:51:26.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-03-25T23:51:26.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-25T23:51:26.562 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-25T23:51:26.562 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:51:26.562 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:51:26.562 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:51:26.563 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:51:26.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-25T23:50:46.304500+0000 2024-03-25T23:51:26.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-25T23:51:27.180 INFO:tasks.workunit.client.0.smithi129.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-25T23:51:27.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-25T23:50:46.304500+0000 2024-03-25T23:51:27.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-25T23:51:27.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-25T23:50:46.304500+0000 2024-03-25T23:51:27.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-25T23:51:27.198 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-25T23:51:27.198 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:51:27.198 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:51:27.198 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:51:27.199 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:51:27.199 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:51:27.199 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:51:27.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:50:46.304500+0000 '>' 2024-03-25T23:50:46.304500+0000 2024-03-25T23:51:27.490 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:51:28.491 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:51:28.491 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:51:28.492 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:51:28.492 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:51:28.492 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:51:28.493 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:51:28.493 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:51:28.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:50:46.304500+0000 '>' 2024-03-25T23:50:46.304500+0000 2024-03-25T23:51:28.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:51:29.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:51:29.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:51:29.803 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:51:29.803 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:51:29.803 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:51:29.803 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:51:29.804 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:51:30.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:50:46.304500+0000 '>' 2024-03-25T23:50:46.304500+0000 2024-03-25T23:51:30.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:51:31.125 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:51:31.125 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:51:31.126 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:51:31.126 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:51:31.126 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:51:31.127 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:51:31.127 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:51:31.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:50:46.304500+0000 '>' 2024-03-25T23:50:46.304500+0000 2024-03-25T23:51:31.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:51:32.437 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:51:32.437 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:51:32.438 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:51:32.438 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:51:32.438 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:51:32.439 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:51:32.439 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:51:32.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:50:46.304500+0000 '>' 2024-03-25T23:50:46.304500+0000 2024-03-25T23:51:32.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:51:33.759 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:51:33.759 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:51:33.760 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:51:33.760 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:51:33.760 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:51:33.761 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:51:33.761 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:51:34.083 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:51:27.219131+0000 '>' 2024-03-25T23:50:46.304500+0000 2024-03-25T23:51:34.083 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-25T23:51:34.083 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-03-25T23:51:34.083 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-25T23:51:34.083 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:51:34.083 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:51:34.084 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 228913"' 2024-03-25T23:51:34.084 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 228913' 2024-03-25T23:51:34.084 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-25T23:51:34.084 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:51:34.084 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:51:34.084 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-25T23:51:34.084 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 228915"' 2024-03-25T23:51:34.084 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 228915' 2024-03-25T23:51:34.084 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-03-25T23:51:34.084 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 228913 228915' 2024-03-25T23:51:34.084 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-25T23:51:34.084 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-25T23:51:34.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 228913 2024-03-25T23:51:34.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/228916: /' 2024-03-25T23:51:34.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-25T23:51:34.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/228918: /' 2024-03-25T23:51:36.228 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:51:36.229 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:51:36.229 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-25T23:51:36.229 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:51:36.229 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-25T23:51:36.229 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:51:36.229 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:51:36.229 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-25T23:51:36.229 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:51:36.229 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-25T23:51:36.229 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:51:36.229 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:51:36.229 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:51:36.229 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:51:36.229 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:51:36.230 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:51:36.230 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-25T23:51:36.230 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:51:36.230 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:51:36.230 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-25T23:51:36.230 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:51:36.230 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-25T23:51:36.230 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-03-25T23:51:36.230 INFO:tasks.workunit.client.0.smithi129.stderr:228916: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-03-25T23:51:36.230 INFO:tasks.workunit.client.0.smithi129.stderr:228916: _ 2024-03-25T23:51:36.230 INFO:tasks.workunit.client.0.smithi129.stderr:228916: hinfo_key 2024-03-25T23:51:36.230 INFO:tasks.workunit.client.0.smithi129.stderr:228916: snapset 2024-03-25T23:51:36.230 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-25T23:51:36.231 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:51:36.231 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:51:36.231 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-25T23:51:36.231 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:51:36.231 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-25T23:51:36.231 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:51:36.234 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:51:36.234 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:51:36.234 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:51:36.234 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-25T23:51:36.234 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-25T23:51:36.234 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:51:36.234 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:51:36.234 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:51:36.234 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:51:36.235 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:51:36.235 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:51:36.235 INFO:tasks.workunit.client.0.smithi129.stderr:228916: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:51:36.235 INFO:tasks.workunit.client.0.smithi129.stderr:228916: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:51:36.235 INFO:tasks.workunit.client.0.smithi129.stderr:228916: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:51:36.235 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:51:36.235 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:51:36.235 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:51:36.235 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:51:36.235 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:51:36.236 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:51:36.236 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:51:36.236 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:51:36.236 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:51:36.236 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:51:36.236 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:51:36.236 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-25T23:51:36.236 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-25T23:51:36.236 INFO:tasks.workunit.client.0.smithi129.stderr:228916: start osd.3 2024-03-25T23:51:36.259 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:51:36.259 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:51:36.259 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-25T23:51:36.259 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:51:36.261 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-25T23:51:36.261 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:51:36.261 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:51:36.261 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-25T23:51:36.261 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:51:36.261 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-25T23:51:36.261 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:51:36.262 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:51:36.262 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:51:36.262 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:51:36.262 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:51:36.262 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:51:36.262 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-25T23:51:36.262 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:51:36.262 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:51:36.262 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-25T23:51:36.262 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:51:36.262 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-25T23:51:36.262 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-03-25T23:51:36.262 INFO:tasks.workunit.client.0.smithi129.stderr:228918: _ 2024-03-25T23:51:36.263 INFO:tasks.workunit.client.0.smithi129.stderr:228918: hinfo_key 2024-03-25T23:51:36.263 INFO:tasks.workunit.client.0.smithi129.stderr:228918: snapset 2024-03-25T23:51:36.263 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-25T23:51:36.263 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:51:36.263 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:51:36.263 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-25T23:51:36.263 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:51:36.263 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-25T23:51:36.263 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:51:36.263 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:51:36.264 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:51:36.264 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:51:36.264 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-25T23:51:36.264 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-25T23:51:36.264 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:51:36.264 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:51:36.264 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:51:36.264 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:51:36.264 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:51:36.264 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:51:36.264 INFO:tasks.workunit.client.0.smithi129.stderr:228918: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:51:36.265 INFO:tasks.workunit.client.0.smithi129.stderr:228918: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:51:36.265 INFO:tasks.workunit.client.0.smithi129.stderr:228918: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:51:36.265 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:51:36.265 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:51:36.265 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:51:36.265 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:51:36.265 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:51:36.265 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:51:36.265 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:51:36.265 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:51:36.265 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:51:36.265 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:51:36.265 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:51:36.265 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-25T23:51:36.266 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-25T23:51:36.266 INFO:tasks.workunit.client.0.smithi129.stderr:228918: start osd.5 2024-03-25T23:51:40.839 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:51:40.839 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-25T23:51:40.839 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-25T23:51:40.839 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:51:40.839 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:51:40.839 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:51:40.839 INFO:tasks.workunit.client.0.smithi129.stderr:228916: 2024-03-25T23:51:36.270+0000 7f29a19647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:51:40.839 INFO:tasks.workunit.client.0.smithi129.stderr:228916: 2024-03-25T23:51:36.274+0000 7f29a19647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:51:40.839 INFO:tasks.workunit.client.0.smithi129.stderr:228916: 2024-03-25T23:51:36.274+0000 7f29a19647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:51:40.839 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-25T23:51:40.840 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:51:40.840 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-25T23:51:40.840 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:51:40.840 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:51:40.840 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:51:40.840 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:51:40.840 INFO:tasks.workunit.client.0.smithi129.stderr:228916: 0 2024-03-25T23:51:40.840 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:51:40.840 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:51:40.840 INFO:tasks.workunit.client.0.smithi129.stderr:228916: 2024-03-25T23:51:36.814+0000 7f29a19647c0 -1 Falling back to public interface 2024-03-25T23:51:40.840 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:51:40.840 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:51:40.840 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:51:40.840 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:51:40.841 INFO:tasks.workunit.client.0.smithi129.stderr:228916: 1 2024-03-25T23:51:40.841 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:51:40.841 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:51:40.841 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:51:40.841 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:51:40.841 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:51:40.841 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:51:40.841 INFO:tasks.workunit.client.0.smithi129.stderr:228916: 2 2024-03-25T23:51:40.841 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:51:40.841 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:51:40.841 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:51:40.841 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:51:40.841 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:51:40.841 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:51:40.842 INFO:tasks.workunit.client.0.smithi129.stderr:228916: 3 2024-03-25T23:51:40.842 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:51:40.842 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:51:40.997 INFO:tasks.workunit.client.0.smithi129.stderr:228916: 2024-03-25T23:51:40.838+0000 7f29aovery_ops 2024-03-25T23:51:40.997 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-25T23:51:40.997 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-25T23:51:40.997 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:51:40.997 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:51:40.997 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:51:40.997 INFO:tasks.workunit.client.0.smithi129.stderr:228918: 2024-03-25T23:51:36.294+0000 7f9ecb97c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:51:40.998 INFO:tasks.workunit.client.0.smithi129.stderr:228918: 2024-03-25T23:51:36.294+0000 7f9ecb97c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:51:40.998 INFO:tasks.workunit.client.0.smithi129.stderr:228918: 2024-03-25T23:51:36.298+0000 7f9ecb97c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:51:40.998 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-25T23:51:40.998 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:51:40.998 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-25T23:51:40.998 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:51:40.998 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:51:40.998 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:51:40.998 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:51:40.998 INFO:tasks.workunit.client.0.smithi129.stderr:228918: 0 2024-03-25T23:51:40.998 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:51:40.998 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:51:40.998 INFO:tasks.workunit.client.0.smithi129.stderr:228918: 2024-03-25T23:51:36.850+0000 7f9ecb97c7c0 -1 Falling back to public interface 2024-03-25T23:51:40.998 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:51:40.999 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:51:40.999 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:51:40.999 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:51:40.999 INFO:tasks.workunit.client.0.smithi129.stderr:228918: 1 2024-03-25T23:51:40.999 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:51:40.999 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:51:40.999 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:51:40.999 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:51:40.999 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:51:40.999 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:51:40.999 INFO:tasks.workunit.client.0.smithi129.stderr:228918: 2 2024-03-25T23:51:40.999 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:51:40.999 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:51:40.999 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:51:40.999 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:51:41.000 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:51:41.000 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:51:41.000 INFO:tasks.workunit.client.0.smithi129.stderr:228918: 3 2024-03-25T23:51:41.000 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:51:41.000 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:51:41.000 INFO:tasks.workunit.client.0.smithi129.stderr:228918: 2024-03-25T23:51:40.878+0000 7f9ecb97c7c0 -1 osd.5 114 log_to_monitors true 2024-03-25T23:51:43.509 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/19647c0 -1 osd.3 113 log_to_monitors true 2024-03-25T23:51:43.509 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:51:43.509 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:51:43.509 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:51:43.509 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:51:43.509 INFO:tasks.workunit.client.0.smithi129.stderr:228916: 4 2024-03-25T23:51:43.509 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:51:43.510 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:51:43.510 INFO:tasks.workunit.client.0.smithi129.stderr:228916: osd.3 up in weight 1 up_from 118 up_thru 118 down_at 114 last_clean_interval [108,113) [v2:127.0.0.1:6826/2998630499,v1:127.0.0.1:6827/2998630499] [v2:127.0.0.1:6828/2998630499,v1:127.0.0.1:6829/2998630499] exists,up 711eac53-c0bb-40a8-a592-40574bbc51ac 2024-03-25T23:51:43.510 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:51:43.510 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:51:43.510 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:51:43.510 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:51:43.510 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:51:43.510 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:51:43.510 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:51:43.510 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:51:43.510 INFO:tasks.workunit.client.0.smithi129.stderr:228916: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:51:43.510 INFO:tasks.workunit.client.0.smithi129.stderr:228916: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:51:43.510 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:51:43.511 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:51:43.511 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:51:43.511 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:51:43.511 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:51:43.511 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:51:43.511 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:51:43.511 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:51:43.511 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:51:43.511 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:51:43.511 INFO:tasks.workunit.client.0.smithi129.stderr:228916: 1 2024-03-25T23:51:43.511 INFO:tasks.workunit.client.0.smithi129.stderr:228916: 2 2024-03-25T23:51:43.511 INFO:tasks.workunit.client.0.smithi129.stderr:228916: 3 2024-03-25T23:51:43.511 INFO:tasks.workunit.client.0.smithi129.stderr:228916: 4 2024-03-25T23:51:43.511 INFO:tasks.workunit.client.0.smithi129.stderr:228916: 5 2024-03-25T23:51:43.511 INFO:tasks.workunit.client.0.smithi129.stderr:228916: 6 2024-03-25T23:51:43.512 INFO:tasks.workunit.client.0.smithi129.stderr:228916: 7 2024-03-25T23:51:43.512 INFO:tasks.workunit.client.0.smithi129.stderr:228916: 8 2024-03-25T23:51:43.512 INFO:tasks.workunit.client.0.smithi129.stderr:228916: 9' 2024-03-25T23:51:43.512 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:51:43.512 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:43.512 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:51:43.512 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836601 2024-03-25T23:51:43.512 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836601 2024-03-25T23:51:43.512 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836601' 2024-03-25T23:51:43.512 INFO:tasks.workunit.client.0.smithi129.stderr:22891ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:51:43.512 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:51:43.512 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:51:43.512 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:51:43.512 INFO:tasks.workunit.client.0.smithi129.stderr:228918: 4 2024-03-25T23:51:43.513 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:51:43.513 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:51:43.513 INFO:tasks.workunit.client.0.smithi129.stderr:228918: osd.5 up in weight 1 up_from 118 up_thru 118 down_at 114 last_clean_interval [111,114) [v2:127.0.0.1:6842/3879546278,v1:127.0.0.1:6843/3879546278] [v2:127.0.0.1:6844/3879546278,v1:127.0.0.1:6845/3879546278] exists,up 35c1c964-6ad5-4a39-91cc-74a5625271e2 2024-03-25T23:51:43.513 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:51:43.513 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:51:43.513 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:51:43.513 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:51:43.513 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:51:43.513 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:51:43.513 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:51:43.513 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:51:43.513 INFO:tasks.workunit.client.0.smithi129.stderr:228918: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:51:43.513 INFO:tasks.workunit.client.0.smithi129.stderr:228918: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:51:43.513 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:51:43.513 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:51:43.514 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:51:43.514 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:51:43.514 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:51:43.514 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:51:43.514 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:51:43.514 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:51:43.514 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:51:43.514 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:51:43.514 INFO:tasks.workunit.client.0.smithi129.stderr:228918: 1 2024-03-25T23:51:43.514 INFO:tasks.workunit.client.0.smithi129.stderr:228918: 2 2024-03-25T23:51:43.514 INFO:tasks.workunit.client.0.smithi129.stderr:228918: 3 2024-03-25T23:51:43.514 INFO:tasks.workunit.client.0.smithi129.stderr:228918: 4 2024-03-25T23:51:43.514 INFO:tasks.workunit.client.0.smithi129.stderr:228918: 5 2024-03-25T23:51:43.514 INFO:tasks.workunit.client.0.smithi129.stderr:228918: 6 2024-03-25T23:51:43.515 INFO:tasks.workunit.client.0.smithi129.stderr:228918: 7 2024-03-25T23:51:43.515 INFO:tasks.workunit.client.0.smithi129.stderr:228918: 8 2024-03-25T23:51:43.515 INFO:tasks.workunit.client.0.smithi129.stderr:228918: 9' 2024-03-25T23:51:43.515 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:51:43.515 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:43.515 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:51:43.515 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836600 2024-03-25T23:51:43.515 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836600 2024-03-25T23:51:43.515 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836600' 2024-03-25T23:51:44.471 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd 6: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:44.471 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:51:44.472 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673075 2024-03-25T23:51:44.472 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673075 2024-03-25T23:51:44.472 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836601 1-42949673075' 2024-03-25T23:51:44.472 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:44.472 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:51:44.472 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509540 2024-03-25T23:51:44.472 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509540 2024-03-25T23:51:44.472 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836601 1-42949673075 2-64424509540' 2024-03-25T23:51:44.472 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:44.472 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:51:44.472 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=506806140931 2024-03-25T23:51:44.472 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 506806140931 2024-03-25T23:51:44.472 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836601 1-42949673075 2-64424509540 3-506806140931' 2024-03-25T23:51:44.472 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:44.473 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T23:51:44.473 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182489 2024-03-25T23:51:44.473 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182489 2024-03-25T23:51:44.473 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836601 1-42949673075 2-64424509540 3-506806140931 4-107374182489' 2024-03-25T23:51:44.473 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:44.473 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T23:51:44.473 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=506806140930 2024-03-25T23:51:44.473 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 506806140930 2024-03-25T23:51:44.473 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836601 1-42949673075 2-64424509540 3-506806140931 4-107374182489 5-506806140930' 2024-03-25T23:51:44.473 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:44.473 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T23:51:44.473 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855430 2024-03-25T23:51:44.473 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855430 2024-03-25T23:51:44.473 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836601 1-42949673075 2-64424509540 3-506806140931 4-107374182489 5-506806140930 6-150323855430' 2024-03-25T23:51:44.474 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:44.613 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: in $ids 2024-03-25T23:51:44.613 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:51:44.613 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673076 2024-03-25T23:51:44.613 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673076 2024-03-25T23:51:44.613 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836600 1-42949673076' 2024-03-25T23:51:44.613 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:44.613 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:51:44.613 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509539 2024-03-25T23:51:44.613 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509539 2024-03-25T23:51:44.613 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836600 1-42949673076 2-64424509539' 2024-03-25T23:51:44.614 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:44.614 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:51:44.614 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=506806140930 2024-03-25T23:51:44.614 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 506806140930 2024-03-25T23:51:44.614 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836600 1-42949673076 2-64424509539 3-506806140930' 2024-03-25T23:51:44.614 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:44.614 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T23:51:44.614 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182490 2024-03-25T23:51:44.614 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182490 2024-03-25T23:51:44.614 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836600 1-42949673076 2-64424509539 3-506806140930 4-107374182490' 2024-03-25T23:51:44.614 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:44.614 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T23:51:44.614 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=506806140931 2024-03-25T23:51:44.615 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 506806140931 2024-03-25T23:51:44.615 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836600 1-42949673076 2-64424509539 3-506806140930 4-107374182490 5-506806140931' 2024-03-25T23:51:44.615 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:44.615 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T23:51:44.615 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855431 2024-03-25T23:51:44.615 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855431 2024-03-25T23:51:44.615 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836600 1-42949673076 2-64424509539 3-506806140930 4-107374182490 5-506806140931 6-150323855431' 2024-03-25T23:51:44.615 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:44.615 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-25T23:51:46.504 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691904 2024-03-25T23:51:46.504 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691904 2024-03-25T23:51:46.504 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836600 1-42949673076 2-64424509539 3-506806140930 4-107374182490 5-506806140931 6-150323855431 7-171798691904' 2024-03-25T23:51:46.504 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:46.505 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-25T23:51:46.505 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528376 2024-03-25T23:51:46.505 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528376 2024-03-25T23:51:46.505 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836600 1-42949673076 2-64424509539 3-506806140930 4-107374182490 5-506806140931 6-150323855431 7-171798691904 8-193273528376' 2024-03-25T23:51:46.505 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:46.505 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-25T23:51:46.505 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=433791696911 2024-03-25T23:51:46.505 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 433791696911 2024-03-25T23:51:46.505 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836600 1-42949673076 2-64424509539 3-506806140930 4-107374182490 5-506806140931 6-150323855431 7-171798691904 8-193273528376 9-433791696911' 2024-03-25T23:51:46.505 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:46.505 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836600 2024-03-25T23:51:46.505 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:46.505 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:51:46.505 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836600 2024-03-25T23:51:46.506 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:46.506 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836600 2024-03-25T23:51:46.506 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836600' 2024-03-25T23:51:46.506 INFO:tasks.workunit.client.0.smithi129.stderr:228918: waiting osd.0 seq 21474836600 2024-03-25T23:51:46.506 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:51:46.506 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836599 -lt 21474836600 2024-03-25T23:51:46.506 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:51:46.506 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:51:46.506 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:51:46.506 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836601 -lt 21474836600 2024-03-25T23:51:46.506 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:46.506 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673076 2024-03-25T23:51:46.506 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:46.506 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:51:46.510 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-4294967307ceph tell osd.7 flush_pg_stats 2024-03-25T23:51:46.510 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691903 2024-03-25T23:51:46.510 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691903 2024-03-25T23:51:46.510 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836601 1-42949673075 2-64424509540 3-506806140931 4-107374182489 5-506806140930 6-150323855430 7-171798691903' 2024-03-25T23:51:46.510 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:46.511 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-25T23:51:46.511 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528377 2024-03-25T23:51:46.511 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528377 2024-03-25T23:51:46.511 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836601 1-42949673075 2-64424509540 3-506806140931 4-107374182489 5-506806140930 6-150323855430 7-171798691903 8-193273528377' 2024-03-25T23:51:46.511 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:51:46.511 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-25T23:51:46.511 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=433791696912 2024-03-25T23:51:46.511 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 433791696912 2024-03-25T23:51:46.511 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836601 1-42949673075 2-64424509540 3-506806140931 4-107374182489 5-506806140930 6-150323855430 7-171798691903 8-193273528377 9-433791696912' 2024-03-25T23:51:46.511 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:46.511 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836601 2024-03-25T23:51:46.511 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:46.511 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:51:46.511 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836601 2024-03-25T23:51:46.512 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:46.512 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836601 2024-03-25T23:51:46.512 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836601' 2024-03-25T23:51:46.512 INFO:tasks.workunit.client.0.smithi129.stderr:228916: waiting osd.0 seq 21474836601 2024-03-25T23:51:46.512 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:51:46.512 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836599 -lt 21474836601 2024-03-25T23:51:46.512 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:51:46.512 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:51:46.512 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:51:46.512 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836601 -lt 21474836601 2024-03-25T23:51:46.512 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:46.513 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673075 2024-03-25T23:51:46.513 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:46.513 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:51:47.512 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/6 2024-03-25T23:51:47.512 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:47.512 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673076 2024-03-25T23:51:47.512 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673076' 2024-03-25T23:51:47.512 INFO:tasks.workunit.client.0.smithi129.stderr:228918: waiting osd.1 seq 42949673076 2024-03-25T23:51:47.512 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:51:47.512 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673076 -lt 42949673076 2024-03-25T23:51:47.512 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:47.512 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509539 2024-03-25T23:51:47.512 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:47.513 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:51:47.513 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509539 2024-03-25T23:51:47.513 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:47.513 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509539 2024-03-25T23:51:47.513 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509539' 2024-03-25T23:51:47.513 INFO:tasks.workunit.client.0.smithi129.stderr:228918: waiting osd.2 seq 64424509539 2024-03-25T23:51:47.513 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:51:47.513 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509541 -lt 64424509539 2024-03-25T23:51:47.513 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:47.513 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-506806140930 2024-03-25T23:51:47.513 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:47.513 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:51:47.513 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-506806140930 2024-03-25T23:51:47.513 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:47.514 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=506806140930 2024-03-25T23:51:47.514 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 506806140930' 2024-03-25T23:51:47.514 INFO:tasks.workunit.client.0.smithi129.stderr:228918: waiting osd.3 seq 506806140930 2024-03-25T23:51:47.514 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:51:47.514 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 506806140932 -lt 506806140930 2024-03-25T23:51:47.514 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:47.514 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182490 2024-03-25T23:51:47.514 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:47.514 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T23:51:47.514 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182490 2024-03-25T23:51:47.514 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:47.514 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182490 2024-03-25T23:51:47.514 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182490' 2024-03-25T23:51:47.514 INFO:tasks.workunit.client.0.smithi129.stderr:228918: waiting osd.4 seq 107374182490 2024-03-25T23:51:47.523 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubunubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673075 2024-03-25T23:51:47.523 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:47.523 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673075 2024-03-25T23:51:47.523 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673075' 2024-03-25T23:51:47.523 INFO:tasks.workunit.client.0.smithi129.stderr:228916: waiting osd.1 seq 42949673075 2024-03-25T23:51:47.523 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:51:47.523 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673076 -lt 42949673075 2024-03-25T23:51:47.523 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:47.523 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509540 2024-03-25T23:51:47.523 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:47.524 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:51:47.524 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509540 2024-03-25T23:51:47.524 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:47.524 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509540 2024-03-25T23:51:47.524 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509540' 2024-03-25T23:51:47.524 INFO:tasks.workunit.client.0.smithi129.stderr:228916: waiting osd.2 seq 64424509540 2024-03-25T23:51:47.524 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:51:47.524 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509541 -lt 64424509540 2024-03-25T23:51:47.524 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:47.524 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-506806140931 2024-03-25T23:51:47.524 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:47.524 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:51:47.524 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-506806140931 2024-03-25T23:51:47.524 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:47.524 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=506806140931 2024-03-25T23:51:47.525 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 506806140931' 2024-03-25T23:51:47.525 INFO:tasks.workunit.client.0.smithi129.stderr:228916: waiting osd.3 seq 506806140931 2024-03-25T23:51:47.525 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:51:47.525 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 506806140932 -lt 506806140931 2024-03-25T23:51:47.525 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:47.525 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:47.525 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182489 2024-03-25T23:51:47.525 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T23:51:47.525 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182489 2024-03-25T23:51:47.525 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:47.525 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182489 2024-03-25T23:51:48.804 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_statu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T23:51:48.804 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182491 -lt 107374182490 2024-03-25T23:51:48.804 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:48.804 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-506806140931 2024-03-25T23:51:48.804 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:48.805 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T23:51:48.805 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-506806140931 2024-03-25T23:51:48.805 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:48.805 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=506806140931 2024-03-25T23:51:48.805 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 506806140931' 2024-03-25T23:51:48.805 INFO:tasks.workunit.client.0.smithi129.stderr:228918: waiting osd.5 seq 506806140931 2024-03-25T23:51:48.805 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T23:51:48.805 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 506806140932 -lt 506806140931 2024-03-25T23:51:48.805 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:48.805 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855431 2024-03-25T23:51:48.805 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:48.805 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-25T23:51:48.805 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855431 2024-03-25T23:51:48.805 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:48.806 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855431 2024-03-25T23:51:48.806 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855431' 2024-03-25T23:51:48.806 INFO:tasks.workunit.client.0.smithi129.stderr:228918: waiting osd.6 seq 150323855431 2024-03-25T23:51:48.806 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-25T23:51:48.806 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855432 -lt 150323855431 2024-03-25T23:51:48.806 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:48.807 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691904 2024-03-25T23:51:48.807 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:48.807 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-25T23:51:48.807 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691904 2024-03-25T23:51:48.807 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:48.807 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691904 2024-03-25T23:51:48.807 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691904' 2024-03-25T23:51:48.807 INFO:tasks.workunit.client.0.smithi129.stderr:228918: waiting osd.7 seq 171798691904 2024-03-25T23:51:48.807 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-25T23:51:48.807 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691905 -lt 171798691904 2024-03-25T23:51:48.807 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:48.958 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/cts: echo 'waiting osd.4 seq 107374182489' 2024-03-25T23:51:48.958 INFO:tasks.workunit.client.0.smithi129.stderr:228916: waiting osd.4 seq 107374182489 2024-03-25T23:51:48.958 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T23:51:48.958 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182491 -lt 107374182489 2024-03-25T23:51:48.958 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:48.958 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-506806140930 2024-03-25T23:51:48.958 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:48.958 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T23:51:48.959 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-506806140930 2024-03-25T23:51:48.959 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:48.959 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=506806140930 2024-03-25T23:51:48.959 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 506806140930' 2024-03-25T23:51:48.959 INFO:tasks.workunit.client.0.smithi129.stderr:228916: waiting osd.5 seq 506806140930 2024-03-25T23:51:48.959 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T23:51:48.959 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 506806140932 -lt 506806140930 2024-03-25T23:51:48.959 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:48.959 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855430 2024-03-25T23:51:48.959 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:48.959 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-25T23:51:48.960 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855430 2024-03-25T23:51:48.960 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:48.960 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855430 2024-03-25T23:51:48.960 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855430' 2024-03-25T23:51:48.960 INFO:tasks.workunit.client.0.smithi129.stderr:228916: waiting osd.6 seq 150323855430 2024-03-25T23:51:48.960 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-25T23:51:48.960 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855432 -lt 150323855430 2024-03-25T23:51:48.960 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:48.960 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691903 2024-03-25T23:51:48.960 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:48.960 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-25T23:51:48.961 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691903 2024-03-25T23:51:48.961 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:48.961 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691903 2024-03-25T23:51:48.961 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691903' 2024-03-25T23:51:48.961 INFO:tasks.workunit.client.0.smithi129.stderr:228916: waiting osd.7 seq 171798691903 2024-03-25T23:51:48.961 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-25T23:51:48.961 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691905 -lt 171798691903 2024-03-25T23:51:50.174 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.seph-helpers.sh:2211: flush_pg_stats: echo 8-193273528376 2024-03-25T23:51:50.174 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:50.174 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-25T23:51:50.174 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528376 2024-03-25T23:51:50.174 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:50.175 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528376 2024-03-25T23:51:50.175 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528376' 2024-03-25T23:51:50.175 INFO:tasks.workunit.client.0.smithi129.stderr:228918: waiting osd.8 seq 193273528376 2024-03-25T23:51:50.175 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-25T23:51:50.175 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528378 -lt 193273528376 2024-03-25T23:51:50.175 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:50.175 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:50.175 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-433791696911 2024-03-25T23:51:50.175 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-25T23:51:50.175 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-433791696911 2024-03-25T23:51:50.175 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:50.175 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=433791696911 2024-03-25T23:51:50.175 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 433791696911' 2024-03-25T23:51:50.175 INFO:tasks.workunit.client.0.smithi129.stderr:228918: waiting osd.9 seq 433791696911 2024-03-25T23:51:50.176 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-25T23:51:50.176 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696912 -lt 433791696911 2024-03-25T23:51:50.176 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:51:50.176 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:51:50.176 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:51:50.176 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:51:50.176 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:51:50.176 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:51:50.176 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:51:50.176 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:51:50.176 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:51:50.176 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:51:50.176 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:51:50.177 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:51:50.177 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:51:50.409 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: h:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:50.409 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528377 2024-03-25T23:51:50.409 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:50.409 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-25T23:51:50.409 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528377 2024-03-25T23:51:50.409 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:50.409 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528377 2024-03-25T23:51:50.409 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528377' 2024-03-25T23:51:50.409 INFO:tasks.workunit.client.0.smithi129.stderr:228916: waiting osd.8 seq 193273528377 2024-03-25T23:51:50.409 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-25T23:51:50.409 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528378 -lt 193273528377 2024-03-25T23:51:50.410 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:51:50.410 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-433791696912 2024-03-25T23:51:50.410 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:51:50.410 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-25T23:51:50.410 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:51:50.410 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-433791696912 2024-03-25T23:51:50.410 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=433791696912 2024-03-25T23:51:50.410 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 433791696912' 2024-03-25T23:51:50.410 INFO:tasks.workunit.client.0.smithi129.stderr:228916: waiting osd.9 seq 433791696912 2024-03-25T23:51:50.410 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-25T23:51:50.410 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696913 -lt 433791696912 2024-03-25T23:51:50.410 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:51:50.410 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:51:50.410 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:51:50.411 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:51:50.411 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:51:50.411 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:51:50.411 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:51:50.411 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:51:50.411 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:51:50.411 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:51:50.411 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:51:50.411 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:51:50.526 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_nu ceph --format json status 2024-03-25T23:51:50.526 INFO:tasks.workunit.client.0.smithi129.stderr:228918: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:51:50.526 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:51:50.526 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:51:50.526 INFO:tasks.workunit.client.0.smithi129.stderr:228918: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:51:50.526 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:51:50.761 INFO:tasks.workunit.client.0.smithi129.stderr:m_pgs 2024-03-25T23:51:50.762 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:51:50.762 INFO:tasks.workunit.client.0.smithi129.stderr:228916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:51:50.762 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:51:50.762 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:51:50.762 INFO:tasks.workunit.client.0.smithi129.stderr:228916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:51:50.762 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:51:50.763 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-25T23:51:50.763 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 228915 2024-03-25T23:51:50.763 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-25T23:51:50.763 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-25T23:51:50.763 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-25T23:51:50.763 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-03-25T23:51:50.764 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-25T23:51:50.764 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-25T23:51:50.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-25T23:51:50.958 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-25T23:51:50.958 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-25T23:51:50.958 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T23:51:50.958 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-25T23:51:50.958 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:51:50.958 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:51:50.959 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:51:50.959 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:51:50.959 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:51:51.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:51:51.281 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T23:51:51.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T23:51:51.283 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T23:51:51.285 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T23:51:51.285 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T23:51:51.285 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T23:51:51.286 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:51:51.286 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T23:51:51.287 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T23:51:51.287 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:51:51.288 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T23:51:51.290 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T23:51:51.290 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-25T23:51:51.362 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T23:51:51.362 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:51:51.362 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:51:51.362 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-25T23:51:51.364 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T23:51:51.365 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T23:51:51.365 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-25T23:51:51.365 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-25T23:51:51.365 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-25T23:51:51.365 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-25T23:51:51.366 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-25T23:51:51.366 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T23:51:51.366 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-25T23:51:51.366 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:51:51.366 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:51:51.367 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:51:51.367 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:51:51.367 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:51:51.367 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:51:51.367 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T23:51:51.368 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T23:51:51.368 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T23:51:51.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T23:51:51.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T23:51:51.370 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T23:51:51.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:51:51.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T23:51:51.372 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T23:51:51.372 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T23:51:51.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T23:51:51.374 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T23:51:51.374 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-25T23:51:51.376 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T23:51:51.376 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:51:51.376 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:51:51.377 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-25T23:51:51.377 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T23:51:51.377 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T23:51:51.377 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-25T23:51:51.378 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-25T23:51:51.378 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:51:51.379 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:51:51.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61636 2024-03-25T23:51:51.381 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-25T23:51:51.381 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-25T23:51:51.381 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-25T23:51:51.381 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-25T23:51:51.381 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_lrc_overwrites td/osd-scrub-repair 2024-03-25T23:51:51.381 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:846: TEST_corrupt_and_repair_lrc_overwrites: '[' true = true ']' 2024-03-25T23:51:51.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_corrupt_and_repair_lrc_overwrites: corrupt_and_repair_lrc td/osd-scrub-repair true 2024-03-25T23:51:51.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:825: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2024-03-25T23:51:51.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:826: corrupt_and_repair_lrc: local allow_overwrites=true 2024-03-25T23:51:51.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:827: corrupt_and_repair_lrc: local poolname=ecpool 2024-03-25T23:51:51.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:829: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2024-03-25T23:51:51.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-25T23:51:51.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-25T23:51:51.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-25T23:51:51.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-25T23:51:51.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-25T23:51:51.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-25T23:51:51.883 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-25T23:51:51.883 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:51:51.883 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:51:51.883 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:51:51.883 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:51:51.883 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:51:51.883 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:51:51.883 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-25T23:51:52.092 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-25T23:51:52.093 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-25T23:51:52.093 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T23:51:52.093 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T23:51:52.093 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-25T23:51:52.094 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-25T23:51:52.094 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T23:51:52.094 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T23:51:52.095 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T23:51:52.095 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T23:51:52.095 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:51:52.095 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:51:52.095 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-25T23:51:52.095 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T23:51:52.095 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get fsid 2024-03-25T23:51:52.212 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-25T23:51:52.212 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T23:51:52.212 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T23:51:52.212 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-25T23:51:52.213 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-25T23:51:52.213 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T23:51:52.213 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T23:51:52.213 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T23:51:52.214 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T23:51:52.214 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:51:52.214 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:51:52.214 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-25T23:51:52.214 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T23:51:52.214 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get mon_host 2024-03-25T23:51:52.324 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:830: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2024-03-25T23:51:52.324 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-25T23:51:52.324 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-25T23:51:52.325 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-25T23:51:52.325 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-25T23:51:52.325 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-25T23:51:52.325 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-25T23:51:53.211 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-25T23:51:53.211 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:51:53.211 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:51:53.212 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:51:53.212 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:51:53.212 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:51:53.212 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:51:53.213 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T23:51:53.214 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T23:51:53.254 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: seq 0 9 2024-03-25T23:51:53.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-25T23:51:53.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 0 2024-03-25T23:51:53.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:51:53.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:51:53.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-25T23:51:53.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:51:53.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-25T23:51:53.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:51:53.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:51:53.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:51:53.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:51:53.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-25T23:51:53.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-25T23:51:53.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:51:53.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:51:53.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:51:53.261 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:51:53.261 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:51:53.262 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:51:53.262 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:51:53.262 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:51:53.262 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:51:53.262 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:51:53.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:51:53.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:51:53.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:51:53.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:51:53.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:51:53.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:51:53.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:51:53.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:51:53.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:51:53.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:51:53.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:51:53.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T23:51:53.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-25T23:51:53.263 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:51:53.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1530ec28-a05f-4444-8c30-2fece967f9df 2024-03-25T23:51:53.264 INFO:tasks.workunit.client.0.smithi129.stdout:add osd0 1530ec28-a05f-4444-8c30-2fece967f9df 2024-03-25T23:51:53.264 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 1530ec28-a05f-4444-8c30-2fece967f9df' 2024-03-25T23:51:53.264 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:51:53.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAZDgJmVAjBEBAAM7lFY6spPQHl9tPi4zty2A== 2024-03-25T23:51:53.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAZDgJmVAjBEBAAM7lFY6spPQHl9tPi4zty2A=="}' 2024-03-25T23:51:53.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1530ec28-a05f-4444-8c30-2fece967f9df -i td/osd-scrub-repair/0/new.json 2024-03-25T23:51:53.529 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:51:53.545 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-25T23:51:53.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAZDgJmVAjBEBAAM7lFY6spPQHl9tPi4zty2A== --osd-uuid 1530ec28-a05f-4444-8c30-2fece967f9df 2024-03-25T23:51:53.580 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:51:53.578+0000 7f10ba3657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:51:53.596 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:51:53.594+0000 7f10ba3657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:51:53.597 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:51:53.594+0000 7f10ba3657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:51:53.598 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:51:53.598+0000 7f10ba3657c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-25T23:51:53.598 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:51:53.598+0000 7f10ba3657c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-25T23:51:53.598 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:51:53.598+0000 7f10ba3657c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-25T23:51:53.696 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:51:53.694+0000 7f10ba3657c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-25T23:52:00.075 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-25T23:52:00.076 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:52:00.078 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd0 key to auth repository 2024-03-25T23:52:00.079 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-25T23:52:00.079 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:52:00.463 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.0 2024-03-25T23:52:00.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-25T23:52:00.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:52:00.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:52:00.465 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:52:00.465 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:52:00.495 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:52:00.498+0000 7f33c2be67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:52:00.530 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:52:00.530+0000 7f33c2be67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:52:00.531 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:52:00.534+0000 7f33c2be67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:52:00.811 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-25T23:52:00.811 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:52:00.811 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T23:52:00.811 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:52:00.811 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:52:00.811 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:52:00.812 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:52:00.812 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:52:00.813 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:52:00.813 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:52:01.085 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:52:01.086+0000 7f33c2be67c0 -1 Falling back to public interface 2024-03-25T23:52:01.161 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:52:02.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:52:02.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:52:02.163 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:52:02.163 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:52:02.163 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:52:02.163 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:52:02.470 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:52:03.471 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:52:03.472 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:52:03.472 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:52:03.472 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:52:03.472 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:52:03.472 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:52:03.709 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:52:03.710+0000 7f33c2be67c0 -1 osd.0 0 log_to_monitors true 2024-03-25T23:52:03.827 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:52:04.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:52:04.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:52:04.830 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:52:04.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:52:04.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:52:04.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:52:05.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:52:06.144 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:52:06.144 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:52:06.144 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:52:06.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:52:06.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:52:06.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:52:06.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:52:07.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:52:07.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:52:07.509 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:52:07.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:52:07.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:52:07.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:52:07.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:52:08.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:52:08.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:52:08.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:52:08.815 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:52:08.815 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:52:08.815 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:52:09.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:52:10.134 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:52:10.134 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:52:10.134 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:52:10.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:52:10.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:52:10.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:52:10.533 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:52:11.535 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:52:11.535 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:52:11.535 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:52:11.536 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:52:11.536 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:52:11.536 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:52:11.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:52:12.862 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:52:12.862 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:52:12.863 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:52:12.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:52:12.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:52:12.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:52:13.174 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:52:14.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:52:14.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:52:14.177 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:52:14.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:52:14.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:52:14.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:52:14.514 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:52:15.516 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:52:15.516 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:52:15.517 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:52:15.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:52:15.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:52:15.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:52:15.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:52:16.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:52:16.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:52:16.835 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:52:16.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:52:16.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:52:16.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:52:17.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:52:18.158 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:52:18.158 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:52:18.158 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:52:18.159 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:52:18.159 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:52:18.159 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:52:18.547 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:52:19.549 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:52:19.549 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:52:19.550 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:52:19.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:52:19.551 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:52:19.551 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:52:19.950 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:52:20.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:52:20.953 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:52:20.953 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:52:20.953 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:52:20.953 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:52:20.953 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:52:21.741 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:52:22.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:52:22.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:52:22.743 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:52:22.744 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:52:22.744 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:52:22.744 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:52:23.296 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:52:24.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:52:24.299 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:52:24.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:52:24.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:52:24.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:52:24.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:52:25.103 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:52:26.105 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:52:26.105 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:52:26.106 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:52:26.106 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:52:26.106 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:52:26.106 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:52:27.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:52:28.018 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:52:28.019 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:52:28.019 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:52:28.019 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:52:28.020 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:52:28.020 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T23:52:28.382 INFO:tasks.workunit.client.0.smithi129.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4170394724,v1:127.0.0.1:6803/4170394724] [v2:127.0.0.1:6804/4170394724,v1:127.0.0.1:6805/4170394724] exists,up 1530ec28-a05f-4444-8c30-2fece967f9df 2024-03-25T23:52:28.385 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:52:28.385 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:52:28.385 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:52:28.385 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-25T23:52:28.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 1 2024-03-25T23:52:28.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:52:28.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:52:28.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-25T23:52:28.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:52:28.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-25T23:52:28.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:52:28.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:52:28.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:52:28.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:52:28.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-25T23:52:28.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-25T23:52:28.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:52:28.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:52:28.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:52:28.387 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:52:28.387 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:52:28.387 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:52:28.387 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:52:28.387 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:52:28.387 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:52:28.387 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:52:28.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:52:28.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:52:28.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:52:28.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:52:28.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:52:28.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:52:28.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:52:28.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:52:28.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:52:28.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:52:28.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:52:28.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T23:52:28.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-25T23:52:28.389 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:52:28.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0535a14c-f350-41ff-9689-ea308fea0d09 2024-03-25T23:52:28.389 INFO:tasks.workunit.client.0.smithi129.stdout:add osd1 0535a14c-f350-41ff-9689-ea308fea0d09 2024-03-25T23:52:28.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 0535a14c-f350-41ff-9689-ea308fea0d09' 2024-03-25T23:52:28.390 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:52:28.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA8DgJmQtaQGBAAFWvEQvitRLuEsoSDHx9IgA== 2024-03-25T23:52:28.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA8DgJmQtaQGBAAFWvEQvitRLuEsoSDHx9IgA=="}' 2024-03-25T23:52:28.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0535a14c-f350-41ff-9689-ea308fea0d09 -i td/osd-scrub-repair/1/new.json 2024-03-25T23:52:28.771 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:52:28.788 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-25T23:52:28.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA8DgJmQtaQGBAAFWvEQvitRLuEsoSDHx9IgA== --osd-uuid 0535a14c-f350-41ff-9689-ea308fea0d09 2024-03-25T23:52:28.822 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:52:28.825+0000 7fbe071d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:52:28.856 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:52:28.857+0000 7fbe071d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:52:28.858 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:52:28.857+0000 7fbe071d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:52:28.858 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:52:28.861+0000 7fbe071d87c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-25T23:52:28.858 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:52:28.861+0000 7fbe071d87c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-25T23:52:28.858 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:52:28.861+0000 7fbe071d87c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-25T23:52:28.929 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:52:28.929+0000 7fbe071d87c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-25T23:52:35.751 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-25T23:52:35.751 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:52:35.753 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd1 key to auth repository 2024-03-25T23:52:35.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-25T23:52:35.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:52:36.129 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.1 2024-03-25T23:52:36.129 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-25T23:52:36.129 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:52:36.130 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:52:36.130 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:52:36.130 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:52:36.160 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:52:36.161+0000 7f90cbd0f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:52:36.163 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:52:36.165+0000 7f90cbd0f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:52:36.164 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:52:36.165+0000 7f90cbd0f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:52:36.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-25T23:52:36.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:52:36.460 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:52:36.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T23:52:36.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:52:36.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:52:36.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:52:36.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:52:36.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:52:36.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:52:36.721 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:52:36.721+0000 7f90cbd0f7c0 -1 Falling back to public interface 2024-03-25T23:52:36.774 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:52:37.775 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:52:37.775 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:52:37.776 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:52:37.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:52:37.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:52:37.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:52:38.082 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:52:39.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:52:39.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:52:39.085 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:52:39.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:52:39.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:52:39.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:52:39.360 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:52:39.361+0000 7f90cbd0f7c0 -1 osd.1 0 log_to_monitors true 2024-03-25T23:52:39.404 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:52:40.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:52:40.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:52:40.406 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:52:40.407 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:52:40.407 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:52:40.407 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:52:40.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:52:41.950 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:52:41.950 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:52:41.951 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:52:41.951 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:52:41.951 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:52:41.951 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:52:42.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:52:43.320 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:52:43.321 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:52:43.321 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:52:43.321 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:52:43.321 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:52:43.322 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:52:43.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:52:44.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:52:44.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:52:44.648 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:52:44.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:52:44.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:52:44.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:52:45.100 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:52:46.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:52:46.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:52:46.102 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:52:46.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:52:46.103 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:52:46.103 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:52:46.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:52:47.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:52:47.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:52:47.510 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:52:47.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:52:47.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:52:47.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:52:47.825 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:52:48.827 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:52:48.827 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:52:48.828 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:52:48.828 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:52:48.828 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:52:48.828 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:52:49.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:52:50.137 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:52:50.137 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:52:50.137 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:52:50.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:52:50.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:52:50.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:52:50.638 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:52:51.639 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:52:51.639 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:52:51.639 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:52:51.640 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:52:51.640 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:52:51.640 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:52:52.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:52:53.115 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:52:53.115 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:52:53.115 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:52:53.116 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:52:53.116 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:52:53.116 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:52:53.490 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:52:54.491 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:52:54.492 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:52:54.492 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:52:54.492 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:52:54.492 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:52:54.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:52:55.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:52:56.114 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:52:56.114 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:52:56.114 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:52:56.115 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:52:56.115 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:52:56.115 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:52:56.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:52:57.549 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:52:57.549 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:52:57.549 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:52:57.549 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:52:57.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:52:57.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:52:58.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:52:59.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:52:59.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:52:59.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:52:59.045 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:52:59.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:52:59.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:52:59.632 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:53:00.633 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:53:00.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:53:00.634 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:53:00.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:53:00.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:53:00.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:53:01.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:53:02.395 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:53:02.396 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:53:02.396 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:53:02.396 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:53:02.396 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:53:02.396 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:53:02.715 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:53:03.717 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:53:03.717 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:53:03.718 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:53:03.718 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:53:03.718 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:53:03.718 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:53:04.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:53:05.242 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:53:05.242 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:53:05.242 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-25T23:53:05.242 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-25T23:53:05.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:53:05.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T23:53:05.562 INFO:tasks.workunit.client.0.smithi129.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4229707319,v1:127.0.0.1:6811/4229707319] [v2:127.0.0.1:6812/4229707319,v1:127.0.0.1:6813/4229707319] exists,up 0535a14c-f350-41ff-9689-ea308fea0d09 2024-03-25T23:53:05.566 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:53:05.566 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:53:05.566 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:53:05.566 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-25T23:53:05.566 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 2 2024-03-25T23:53:05.566 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:53:05.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:53:05.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-25T23:53:05.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:53:05.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-25T23:53:05.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:53:05.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:53:05.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:53:05.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:53:05.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-25T23:53:05.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-25T23:53:05.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:53:05.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:53:05.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:53:05.567 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:53:05.568 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:53:05.568 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:53:05.568 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:53:05.568 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:53:05.568 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:53:05.568 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:53:05.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:53:05.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:53:05.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:53:05.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:53:05.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:53:05.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:53:05.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:53:05.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:53:05.569 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:53:05.569 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:53:05.569 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:53:05.569 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T23:53:05.569 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-25T23:53:05.569 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:53:05.569 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e668e746-230c-4bce-a4ab-f706983be985 2024-03-25T23:53:05.570 INFO:tasks.workunit.client.0.smithi129.stdout:add osd2 e668e746-230c-4bce-a4ab-f706983be985 2024-03-25T23:53:05.570 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 e668e746-230c-4bce-a4ab-f706983be985' 2024-03-25T23:53:05.570 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:53:05.591 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBhDgJmpEpZIxAAjWkl9dIrtwXg3yRRFPa0rQ== 2024-03-25T23:53:05.591 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBhDgJmpEpZIxAAjWkl9dIrtwXg3yRRFPa0rQ=="}' 2024-03-25T23:53:05.591 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e668e746-230c-4bce-a4ab-f706983be985 -i td/osd-scrub-repair/2/new.json 2024-03-25T23:53:05.956 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:53:05.974 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-25T23:53:05.975 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBhDgJmpEpZIxAAjWkl9dIrtwXg3yRRFPa0rQ== --osd-uuid e668e746-230c-4bce-a4ab-f706983be985 2024-03-25T23:53:06.010 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:53:06.012+0000 7f1817f3d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:53:06.058 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:53:06.056+0000 7f1817f3d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:53:06.060 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:53:06.060+0000 7f1817f3d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:53:06.060 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:53:06.060+0000 7f1817f3d7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-25T23:53:06.060 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:53:06.060+0000 7f1817f3d7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-25T23:53:06.060 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:53:06.060+0000 7f1817f3d7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-25T23:53:06.133 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:53:06.132+0000 7f1817f3d7c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-25T23:53:13.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-25T23:53:13.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:53:13.300 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd2 key to auth repository 2024-03-25T23:53:13.300 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-25T23:53:13.300 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:53:13.701 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.2 2024-03-25T23:53:13.701 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-25T23:53:13.702 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:53:13.702 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:53:13.702 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:53:13.702 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:53:13.733 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:53:13.732+0000 7f85492297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:53:13.765 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:53:13.764+0000 7f85492297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:53:13.767 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:53:13.768+0000 7f85492297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:53:14.064 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-25T23:53:14.064 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:53:14.064 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-25T23:53:14.064 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:53:14.064 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:53:14.064 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:53:14.065 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:53:14.065 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:53:14.065 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:53:14.065 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:53:14.320 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:53:14.320+0000 7f85492297c0 -1 Falling back to public interface 2024-03-25T23:53:14.425 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:53:15.427 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:53:15.428 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:53:15.428 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:53:15.428 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:53:15.428 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:53:15.429 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:53:15.748 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:53:16.750 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:53:16.750 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:53:16.750 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:53:16.750 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:53:16.751 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:53:16.751 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:53:16.934 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:53:16.936+0000 7f85492297c0 -1 osd.2 0 log_to_monitors true 2024-03-25T23:53:17.072 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:53:18.075 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:53:18.075 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:53:18.075 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:53:18.076 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:53:18.076 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:53:18.076 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:53:18.416 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:53:19.417 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:53:19.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:53:19.418 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:53:19.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:53:19.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:53:19.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:53:19.880 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:53:20.882 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:53:20.882 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:53:20.882 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:53:20.883 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:53:20.883 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:53:20.883 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:53:21.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:53:22.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:53:22.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:53:22.278 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:53:22.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:53:22.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:53:22.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:53:22.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:53:23.647 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:53:23.647 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:53:23.647 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:53:23.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:53:23.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:53:23.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:53:24.219 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:53:25.221 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:53:25.221 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:53:25.221 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:53:25.222 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:53:25.222 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:53:25.222 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:53:25.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:53:26.534 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:53:26.534 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:53:26.534 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:53:26.534 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:53:26.535 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:53:26.535 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:53:26.995 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:53:27.997 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:53:27.997 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:53:27.998 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:53:27.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:53:27.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:53:27.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:53:28.315 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:53:29.316 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:53:29.316 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:53:29.317 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:53:29.317 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:53:29.317 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:53:29.317 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:53:29.684 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:53:30.685 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:53:30.685 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:53:30.686 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:53:30.686 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:53:30.686 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:53:30.686 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:53:31.133 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:53:32.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:53:32.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:53:32.135 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:53:32.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:53:32.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:53:32.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:53:32.786 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:53:33.788 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:53:33.788 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:53:33.788 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:53:33.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:53:33.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:53:33.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:53:34.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:53:35.582 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:53:35.582 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:53:35.582 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:53:35.583 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:53:35.583 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:53:35.583 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:53:36.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:53:37.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:53:37.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:53:37.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:53:37.042 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:53:37.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:53:37.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:53:37.403 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:53:38.404 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:53:38.405 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:53:38.405 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:53:38.405 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:53:38.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:53:38.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:53:39.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:53:40.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:53:40.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:53:40.200 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:53:40.200 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:53:40.200 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:53:40.200 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:53:40.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:53:41.614 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:53:41.614 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:53:41.614 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:53:41.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:53:41.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:53:41.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:53:42.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:53:43.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:53:43.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:53:43.344 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-25T23:53:43.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-25T23:53:43.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:53:43.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T23:53:43.756 INFO:tasks.workunit.client.0.smithi129.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3908688614,v1:127.0.0.1:6819/3908688614] [v2:127.0.0.1:6820/3908688614,v1:127.0.0.1:6821/3908688614] exists,up e668e746-230c-4bce-a4ab-f706983be985 2024-03-25T23:53:43.759 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:53:43.759 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:53:43.759 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:53:43.759 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-25T23:53:43.759 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 3 2024-03-25T23:53:43.759 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:53:43.759 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:53:43.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-03-25T23:53:43.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:53:43.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-03-25T23:53:43.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:53:43.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:53:43.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:53:43.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:53:43.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-25T23:53:43.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-25T23:53:43.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:53:43.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:53:43.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:53:43.760 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:53:43.760 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:53:43.761 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:53:43.761 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:53:43.761 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:53:43.761 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:53:43.761 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:53:43.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:53:43.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:53:43.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:53:43.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:53:43.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:53:43.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:53:43.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:53:43.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:53:43.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:53:43.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:53:43.762 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:53:43.762 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T23:53:43.762 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-03-25T23:53:43.762 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:53:43.762 INFO:tasks.workunit.client.0.smithi129.stdout:add osd3 25e4ca68-13d2-4df8-9930-c8c629bba9ef 2024-03-25T23:53:43.763 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=25e4ca68-13d2-4df8-9930-c8c629bba9ef 2024-03-25T23:53:43.763 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 25e4ca68-13d2-4df8-9930-c8c629bba9ef' 2024-03-25T23:53:43.763 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:53:43.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCHDgJmX6njLhAAP2QwDE81OdmJNBpIaujo4Q== 2024-03-25T23:53:43.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCHDgJmX6njLhAAP2QwDE81OdmJNBpIaujo4Q=="}' 2024-03-25T23:53:43.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 25e4ca68-13d2-4df8-9930-c8c629bba9ef -i td/osd-scrub-repair/3/new.json 2024-03-25T23:53:44.311 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:53:44.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-03-25T23:53:44.339 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCHDgJmX6njLhAAP2QwDE81OdmJNBpIaujo4Q== --osd-uuid 25e4ca68-13d2-4df8-9930-c8c629bba9ef 2024-03-25T23:53:44.374 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:53:44.371+0000 7f0aa2bdb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:53:44.404 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:53:44.403+0000 7f0aa2bdb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:53:44.406 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:53:44.403+0000 7f0aa2bdb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:53:44.406 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:53:44.407+0000 7f0aa2bdb7c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-25T23:53:44.406 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:53:44.407+0000 7f0aa2bdb7c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-25T23:53:44.406 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:53:44.407+0000 7f0aa2bdb7c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-25T23:53:44.474 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:53:44.471+0000 7f0aa2bdb7c0 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-03-25T23:53:51.324 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-03-25T23:53:51.324 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:53:51.325 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd3 key to auth repository 2024-03-25T23:53:51.325 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-03-25T23:53:51.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:53:51.719 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-03-25T23:53:51.719 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.3 2024-03-25T23:53:51.720 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:53:51.720 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:53:51.720 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:53:51.720 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:53:51.742 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:53:51.743+0000 7f77f7d1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:53:51.744 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:53:51.743+0000 7f77f7d1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:53:51.745 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:53:51.743+0000 7f77f7d1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:53:52.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-03-25T23:53:52.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:53:52.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-25T23:53:52.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:53:52.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:53:52.091 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:53:52.091 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:53:52.091 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:53:52.091 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:53:52.091 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:53:52.288 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:53:52.287+0000 7f77f7d1d7c0 -1 Falling back to public interface 2024-03-25T23:53:52.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:53:53.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:53:53.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:53:53.433 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:53:53.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:53:53.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:53:53.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:53:53.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:53:54.747 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:53:54.747 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:53:54.748 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:53:54.748 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:53:54.748 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:53:54.748 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:53:54.962 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:53:54.959+0000 7f77f7d1d7c0 -1 osd.3 0 log_to_monitors true 2024-03-25T23:53:55.083 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:53:56.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:53:56.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:53:56.086 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:53:56.087 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:53:56.087 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:53:56.087 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:53:56.443 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:53:57.445 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:53:57.445 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:53:57.445 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:53:57.446 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:53:57.446 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:53:57.446 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:53:57.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:53:58.763 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:53:58.763 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:53:58.764 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:53:58.764 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:53:58.764 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:53:58.764 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:53:59.170 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:54:00.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:54:00.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:54:00.172 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:54:00.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:54:00.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:54:00.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:54:00.655 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:54:01.658 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:54:01.658 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:54:01.658 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:54:01.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:54:01.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:54:01.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:54:02.184 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:54:03.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:54:03.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:54:03.186 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:54:03.187 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:54:03.187 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:54:03.187 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:54:03.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:54:04.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:54:04.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:54:04.507 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:54:04.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:54:04.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:54:04.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:54:04.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:54:05.916 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:54:05.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:54:05.917 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:54:05.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:54:05.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:54:05.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:54:06.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:54:07.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:54:07.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:54:07.230 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:54:07.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:54:07.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:54:07.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:54:07.549 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:54:08.551 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:54:08.551 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:54:08.551 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:54:08.552 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:54:08.552 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:54:08.552 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:54:08.912 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:54:09.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:54:09.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:54:09.914 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:54:09.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:54:09.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:54:09.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:54:10.857 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:54:11.859 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:54:11.859 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:54:11.859 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:54:11.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:54:11.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:54:11.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:54:12.934 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:54:13.933 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:54:13.934 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:54:13.934 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:54:13.934 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:54:13.935 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:54:13.935 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:54:14.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:54:15.862 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:54:15.862 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:54:15.862 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:54:15.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:54:15.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:54:15.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:54:16.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:54:17.384 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:54:17.384 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:54:17.384 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:54:17.385 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:54:17.385 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:54:17.385 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:54:18.110 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:54:19.111 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:54:19.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:54:19.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:54:19.112 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:54:19.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:54:19.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:54:19.774 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:54:20.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:54:20.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:54:20.777 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:54:20.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:54:20.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:54:20.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:54:21.091 INFO:tasks.workunit.client.0.smithi129.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3356026864,v1:127.0.0.1:6827/3356026864] [v2:127.0.0.1:6828/3356026864,v1:127.0.0.1:6829/3356026864] exists,up 25e4ca68-13d2-4df8-9930-c8c629bba9ef 2024-03-25T23:54:21.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:54:21.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:54:21.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:54:21.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-25T23:54:21.094 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 4 2024-03-25T23:54:21.094 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:54:21.094 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:54:21.094 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-03-25T23:54:21.094 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:54:21.094 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/4 2024-03-25T23:54:21.094 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:54:21.094 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:54:21.094 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:54:21.094 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:54:21.094 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2024-03-25T23:54:21.094 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2024-03-25T23:54:21.094 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:54:21.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:54:21.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:54:21.095 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:54:21.095 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:54:21.095 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:54:21.095 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:54:21.095 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:54:21.095 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:54:21.095 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:54:21.096 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:54:21.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:54:21.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:54:21.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:54:21.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:54:21.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:54:21.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:54:21.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:54:21.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:54:21.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:54:21.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:54:21.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T23:54:21.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/4 2024-03-25T23:54:21.097 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:54:21.098 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5ef2bee6-8529-4d40-bc52-bfc962e2e17b 2024-03-25T23:54:21.098 INFO:tasks.workunit.client.0.smithi129.stdout:add osd4 5ef2bee6-8529-4d40-bc52-bfc962e2e17b 2024-03-25T23:54:21.098 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 5ef2bee6-8529-4d40-bc52-bfc962e2e17b' 2024-03-25T23:54:21.099 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:54:21.114 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCtDgJmjDH3BhAAZfIUcx6Mrb/lFKifQi8aZg== 2024-03-25T23:54:21.114 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCtDgJmjDH3BhAAZfIUcx6Mrb/lFKifQi8aZg=="}' 2024-03-25T23:54:21.114 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5ef2bee6-8529-4d40-bc52-bfc962e2e17b -i td/osd-scrub-repair/4/new.json 2024-03-25T23:54:21.575 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:54:21.597 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/4/new.json 2024-03-25T23:54:21.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCtDgJmjDH3BhAAZfIUcx6Mrb/lFKifQi8aZg== --osd-uuid 5ef2bee6-8529-4d40-bc52-bfc962e2e17b 2024-03-25T23:54:21.635 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:54:21.634+0000 7f1c6b7b67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:54:21.711 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:54:21.710+0000 7f1c6b7b67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:54:21.713 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:54:21.710+0000 7f1c6b7b67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:54:21.713 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:54:21.710+0000 7f1c6b7b67c0 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-03-25T23:54:21.713 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:54:21.710+0000 7f1c6b7b67c0 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-03-25T23:54:21.713 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:54:21.710+0000 7f1c6b7b67c0 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-03-25T23:54:21.794 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:54:21.790+0000 7f1c6b7b67c0 -1 bluestore(td/osd-scrub-repair/4) _read_fsid unparsable uuid 2024-03-25T23:54:28.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/4/keyring 2024-03-25T23:54:28.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:54:28.802 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd4 key to auth repository 2024-03-25T23:54:28.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-03-25T23:54:28.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:54:29.198 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.4 2024-03-25T23:54:29.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-03-25T23:54:29.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:54:29.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:54:29.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:54:29.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:54:29.232 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:54:29.230+0000 7f59acd607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:54:29.269 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:54:29.266+0000 7f59acd607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:54:29.271 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:54:29.270+0000 7f59acd607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:54:29.526 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-03-25T23:54:29.526 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:54:29.526 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:54:29.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-03-25T23:54:29.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:54:29.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:54:29.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:54:29.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:54:29.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:54:29.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:54:29.824 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:54:29.822+0000 7f59acd607c0 -1 Falling back to public interface 2024-03-25T23:54:29.849 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:54:30.851 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:54:30.851 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:54:30.851 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:54:30.852 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:54:30.852 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:54:30.852 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:54:31.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:54:32.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:54:32.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:54:32.176 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:54:32.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:54:32.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:54:32.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:54:32.525 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:54:32.522+0000 7f59acd607c0 -1 osd.4 0 log_to_monitors true 2024-03-25T23:54:32.545 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:54:33.547 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:54:33.547 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:54:33.548 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:54:33.548 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:54:33.548 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:54:33.548 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:54:33.882 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:54:34.883 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:54:34.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:54:34.884 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:54:34.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:54:34.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:54:34.885 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:54:35.204 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:54:36.206 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:54:36.206 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:54:36.206 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:54:36.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:54:36.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:54:36.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:54:36.619 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:54:37.621 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:54:37.621 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:54:37.621 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:54:37.622 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:54:37.622 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:54:37.622 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:54:37.943 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:54:38.945 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:54:38.945 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:54:38.945 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:54:38.945 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:54:38.945 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:54:38.946 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:54:39.383 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:54:40.384 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:54:40.385 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:54:40.385 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:54:40.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:54:40.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:54:40.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:54:40.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:54:41.705 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:54:41.705 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:54:41.705 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:54:41.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:54:41.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:54:41.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:54:42.050 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:54:43.051 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:54:43.052 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:54:43.052 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:54:43.052 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:54:43.052 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:54:43.053 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:54:43.408 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:54:44.411 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:54:44.411 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:54:44.411 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:54:44.412 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:54:44.412 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:54:44.412 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:54:44.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:54:45.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:54:45.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:54:45.784 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:54:45.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:54:45.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:54:45.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:54:46.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:54:47.126 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:54:47.126 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:54:47.126 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:54:47.126 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:54:47.127 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:54:47.127 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:54:47.905 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:54:48.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:54:48.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:54:48.907 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:54:48.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:54:48.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:54:48.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:54:49.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:54:50.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:54:50.589 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:54:50.589 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:54:50.589 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:54:50.589 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:54:50.590 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:54:51.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:54:52.088 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:54:52.088 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:54:52.088 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:54:52.089 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:54:52.089 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:54:52.089 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:54:52.818 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:54:53.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:54:53.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:54:53.820 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:54:53.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:54:53.821 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:54:53.821 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:54:54.638 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:54:55.640 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:54:55.640 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:54:55.641 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:54:55.641 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:54:55.641 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:54:55.641 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:54:56.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:54:57.512 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:54:57.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:54:57.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:54:57.513 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:54:57.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:54:57.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:54:57.851 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:54:58.853 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:54:58.853 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:54:58.853 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-25T23:54:58.853 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-25T23:54:58.854 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:54:58.854 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T23:54:59.172 INFO:tasks.workunit.client.0.smithi129.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/379880571,v1:127.0.0.1:6835/379880571] [v2:127.0.0.1:6836/379880571,v1:127.0.0.1:6837/379880571] exists,up 5ef2bee6-8529-4d40-bc52-bfc962e2e17b 2024-03-25T23:54:59.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:54:59.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:54:59.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:54:59.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-25T23:54:59.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 5 2024-03-25T23:54:59.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:54:59.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:54:59.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-03-25T23:54:59.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:54:59.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/5 2024-03-25T23:54:59.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:54:59.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:54:59.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:54:59.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:54:59.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-25T23:54:59.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-25T23:54:59.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:54:59.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:54:59.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:54:59.176 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:54:59.176 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:54:59.176 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:54:59.176 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:54:59.177 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:54:59.177 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:54:59.177 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:54:59.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:54:59.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:54:59.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:54:59.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:54:59.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:54:59.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:54:59.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:54:59.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:54:59.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:54:59.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:54:59.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:54:59.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T23:54:59.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/5 2024-03-25T23:54:59.180 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:54:59.180 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=67160811-ca71-46db-8586-da42b099b746 2024-03-25T23:54:59.180 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 67160811-ca71-46db-8586-da42b099b746' 2024-03-25T23:54:59.180 INFO:tasks.workunit.client.0.smithi129.stdout:add osd5 67160811-ca71-46db-8586-da42b099b746 2024-03-25T23:54:59.180 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:54:59.202 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDTDgJmLGo0DBAAtdX6ayrntCH7LtmC6ITjkw== 2024-03-25T23:54:59.202 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDTDgJmLGo0DBAAtdX6ayrntCH7LtmC6ITjkw=="}' 2024-03-25T23:54:59.202 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 67160811-ca71-46db-8586-da42b099b746 -i td/osd-scrub-repair/5/new.json 2024-03-25T23:54:59.601 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:54:59.620 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/5/new.json 2024-03-25T23:54:59.621 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDTDgJmLGo0DBAAtdX6ayrntCH7LtmC6ITjkw== --osd-uuid 67160811-ca71-46db-8586-da42b099b746 2024-03-25T23:54:59.656 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:54:59.657+0000 7f4f237b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:54:59.703 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:54:59.705+0000 7f4f237b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:54:59.705 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:54:59.705+0000 7f4f237b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:54:59.705 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:54:59.705+0000 7f4f237b27c0 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-03-25T23:54:59.706 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:54:59.705+0000 7f4f237b27c0 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-03-25T23:54:59.706 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:54:59.705+0000 7f4f237b27c0 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-03-25T23:54:59.793 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:54:59.793+0000 7f4f237b27c0 -1 bluestore(td/osd-scrub-repair/5) _read_fsid unparsable uuid 2024-03-25T23:55:07.109 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/5/keyring 2024-03-25T23:55:07.109 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:55:07.110 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd5 key to auth repository 2024-03-25T23:55:07.111 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-03-25T23:55:07.111 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:55:07.506 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.5 2024-03-25T23:55:07.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-03-25T23:55:07.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:55:07.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:55:07.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:55:07.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:55:07.538 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:55:07.537+0000 7ff19b4b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:55:07.570 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:55:07.569+0000 7ff19b4b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:55:07.571 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:55:07.573+0000 7ff19b4b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:55:07.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-03-25T23:55:07.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:55:07.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-25T23:55:07.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:55:07.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:55:07.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:55:07.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:55:07.830 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:55:07.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:55:07.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:55:08.120 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:55:08.121+0000 7ff19b4b47c0 -1 Falling back to public interface 2024-03-25T23:55:08.158 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:55:09.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:55:09.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:55:09.160 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:55:09.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:55:09.161 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:55:09.161 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:55:09.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:55:10.463 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:55:10.463 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:55:10.463 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:55:10.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:55:10.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:55:10.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:55:10.787 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:55:10.867 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:55:10.869+0000 7ff19b4b47c0 -1 osd.5 0 log_to_monitors true 2024-03-25T23:55:11.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:55:11.789 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:55:11.790 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:55:11.790 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:55:11.790 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:55:11.790 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:55:12.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:55:13.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:55:13.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:55:13.179 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:55:13.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:55:13.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:55:13.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:55:13.502 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:55:14.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:55:14.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:55:14.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:55:14.505 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:55:14.506 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:55:14.506 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:55:14.902 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:55:15.903 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:55:15.904 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:55:15.904 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:55:15.904 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:55:15.905 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:55:15.905 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:55:16.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:55:17.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:55:17.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:55:17.230 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:55:17.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:55:17.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:55:17.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:55:17.552 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:55:18.554 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:55:18.554 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:55:18.555 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:55:18.555 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:55:18.555 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:55:18.555 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:55:18.883 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:55:19.885 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:55:19.885 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:55:19.886 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:55:19.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:55:19.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:55:19.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:55:20.223 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:55:21.224 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:55:21.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:55:21.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:55:21.225 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:55:21.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:55:21.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:55:21.544 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:55:22.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:55:22.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:55:22.547 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:55:22.547 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:55:22.547 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:55:22.547 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:55:23.022 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:55:24.024 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:55:24.024 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:55:24.024 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:55:24.025 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:55:24.025 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:55:24.025 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:55:24.335 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:55:25.338 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:55:25.338 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:55:25.338 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:55:25.339 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:55:25.339 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:55:25.339 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:55:25.808 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:55:26.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:55:26.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:55:26.811 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:55:26.811 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:55:26.811 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:55:26.811 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:55:27.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:55:28.372 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:55:28.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:55:28.373 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:55:28.374 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:55:28.374 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:55:28.374 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:55:29.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:55:30.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:55:30.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:55:30.256 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:55:30.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:55:30.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:55:30.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:55:31.174 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:55:32.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:55:32.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:55:32.176 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:55:32.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:55:32.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:55:32.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:55:32.822 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:55:33.824 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:55:33.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:55:33.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:55:33.873 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:55:33.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:55:33.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:55:34.726 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:55:35.728 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:55:35.728 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:55:35.728 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:55:35.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:55:35.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:55:35.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:55:36.324 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:55:37.327 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:55:37.327 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:55:37.327 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-25T23:55:37.328 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-25T23:55:37.328 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:55:37.328 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:55:37.647 INFO:tasks.workunit.client.0.smithi129.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/1880479322,v1:127.0.0.1:6843/1880479322] [v2:127.0.0.1:6844/1880479322,v1:127.0.0.1:6845/1880479322] exists,up 67160811-ca71-46db-8586-da42b099b746 2024-03-25T23:55:37.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:55:37.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:55:37.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:55:37.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-25T23:55:37.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 6 2024-03-25T23:55:37.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:55:37.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:55:37.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-03-25T23:55:37.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:55:37.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/6 2024-03-25T23:55:37.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:55:37.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:55:37.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:55:37.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:55:37.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2024-03-25T23:55:37.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2024-03-25T23:55:37.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:55:37.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:55:37.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:55:37.651 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:55:37.651 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:55:37.651 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:55:37.652 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:55:37.652 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:55:37.653 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:55:37.653 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:55:37.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:55:37.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:55:37.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:55:37.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:55:37.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:55:37.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:55:37.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:55:37.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:55:37.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:55:37.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:55:37.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:55:37.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T23:55:37.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/6 2024-03-25T23:55:37.654 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:55:37.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=173531e3-f1e3-4319-a5cb-a603e3a0c551 2024-03-25T23:55:37.655 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 173531e3-f1e3-4319-a5cb-a603e3a0c551' 2024-03-25T23:55:37.655 INFO:tasks.workunit.client.0.smithi129.stdout:add osd6 173531e3-f1e3-4319-a5cb-a603e3a0c551 2024-03-25T23:55:37.655 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:55:37.677 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD5DgJmWid6KBAAxeKX8yFqaiI2w90dgjkZ5A== 2024-03-25T23:55:37.677 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD5DgJmWid6KBAAxeKX8yFqaiI2w90dgjkZ5A=="}' 2024-03-25T23:55:37.677 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 173531e3-f1e3-4319-a5cb-a603e3a0c551 -i td/osd-scrub-repair/6/new.json 2024-03-25T23:55:38.090 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:55:38.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/6/new.json 2024-03-25T23:55:38.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD5DgJmWid6KBAAxeKX8yFqaiI2w90dgjkZ5A== --osd-uuid 173531e3-f1e3-4319-a5cb-a603e3a0c551 2024-03-25T23:55:38.148 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:55:38.149+0000 7f7dc393c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:55:38.200 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:55:38.201+0000 7f7dc393c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:55:38.201 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:55:38.201+0000 7f7dc393c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:55:38.202 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:55:38.201+0000 7f7dc393c7c0 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-03-25T23:55:38.202 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:55:38.201+0000 7f7dc393c7c0 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-03-25T23:55:38.202 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:55:38.201+0000 7f7dc393c7c0 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-03-25T23:55:38.308 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:55:38.309+0000 7f7dc393c7c0 -1 bluestore(td/osd-scrub-repair/6) _read_fsid unparsable uuid 2024-03-25T23:55:44.155 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/6/keyring 2024-03-25T23:55:44.155 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:55:44.157 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd6 key to auth repository 2024-03-25T23:55:44.158 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-03-25T23:55:44.158 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:55:44.535 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.6 2024-03-25T23:55:44.536 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-03-25T23:55:44.536 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:55:44.536 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:55:44.537 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:55:44.537 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:55:44.568 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:55:44.568+0000 7f2a429357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:55:44.570 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:55:44.572+0000 7f2a429357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:55:44.571 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:55:44.572+0000 7f2a429357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:55:44.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-03-25T23:55:44.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:55:44.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-03-25T23:55:44.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:55:44.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:55:44.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:55:44.874 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:55:44.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:55:44.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:55:44.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:55:45.124 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:55:45.124+0000 7f2a429357c0 -1 Falling back to public interface 2024-03-25T23:55:45.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:55:46.208 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:55:46.208 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:55:46.209 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:55:46.209 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:55:46.209 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:55:46.209 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:55:46.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:55:47.529 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:55:47.529 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:55:47.529 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:55:47.530 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:55:47.530 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:55:47.530 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:55:47.573 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:55:47.572+0000 7f2a429357c0 -1 osd.6 0 log_to_monitors true 2024-03-25T23:55:47.913 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:55:48.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:55:48.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:55:48.916 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:55:48.916 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:55:48.916 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:55:48.916 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:55:49.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:55:50.242 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:55:50.242 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:55:50.242 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:55:50.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:55:50.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:55:50.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:55:50.813 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:55:51.815 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:55:51.815 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:55:51.816 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:55:51.816 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:55:51.816 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:55:51.816 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:55:52.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:55:53.141 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:55:53.141 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:55:53.141 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:55:53.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:55:53.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:55:53.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:55:53.482 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:55:54.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:55:54.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:55:54.484 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:55:54.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:55:54.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:55:54.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:55:54.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:55:55.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:55:55.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:55:55.872 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:55:55.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:55:55.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:55:55.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:55:56.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:55:57.209 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:55:57.210 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:55:57.210 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:55:57.211 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:55:57.211 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:55:57.211 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:55:57.638 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:55:58.640 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:55:58.641 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:55:58.641 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:55:58.641 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:55:58.641 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:55:58.642 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:55:58.978 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:55:59.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:55:59.980 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:55:59.980 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:55:59.980 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:55:59.980 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:55:59.980 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:56:00.302 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:56:01.304 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:56:01.304 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:56:01.305 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:56:01.305 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:56:01.305 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:56:01.305 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:56:01.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:56:02.673 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:56:02.673 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:56:02.673 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:56:02.673 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:56:02.674 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:56:02.674 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:56:03.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:56:04.104 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:56:04.105 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:56:04.105 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:56:04.105 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:56:04.105 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:56:04.106 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:56:04.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:56:05.921 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:56:05.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:56:05.922 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:56:05.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:56:05.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:56:05.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:56:06.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:56:07.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:56:07.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:56:07.834 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:56:07.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:56:07.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:56:07.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:56:08.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:56:09.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:56:09.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:56:09.436 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:56:09.436 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:56:09.436 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:56:09.436 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:56:10.423 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:56:11.424 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:56:11.424 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:56:11.424 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:56:11.425 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:56:11.425 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:56:11.425 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:56:12.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:56:13.436 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:56:13.437 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:56:13.437 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:56:13.438 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:56:13.438 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:56:13.438 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:56:13.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:56:14.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:56:14.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:56:14.874 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-25T23:56:14.875 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-25T23:56:14.875 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:56:14.876 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T23:56:15.197 INFO:tasks.workunit.client.0.smithi129.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/440146549,v1:127.0.0.1:6851/440146549] [v2:127.0.0.1:6852/440146549,v1:127.0.0.1:6853/440146549] exists,up 173531e3-f1e3-4319-a5cb-a603e3a0c551 2024-03-25T23:56:15.200 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:56:15.200 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:56:15.200 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:56:15.200 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-25T23:56:15.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 7 2024-03-25T23:56:15.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:56:15.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:56:15.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-03-25T23:56:15.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:56:15.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/7 2024-03-25T23:56:15.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:56:15.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:56:15.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:56:15.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:56:15.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2024-03-25T23:56:15.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2024-03-25T23:56:15.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:56:15.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:56:15.202 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:56:15.202 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:56:15.202 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:56:15.202 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:56:15.202 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:56:15.202 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:56:15.202 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:56:15.202 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:56:15.202 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:56:15.202 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:56:15.202 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:56:15.202 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:56:15.202 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:56:15.203 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:56:15.203 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:56:15.203 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:56:15.203 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:56:15.203 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:56:15.203 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:56:15.203 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T23:56:15.203 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/7 2024-03-25T23:56:15.204 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:56:15.204 INFO:tasks.workunit.client.0.smithi129.stdout:add osd7 6bee79ee-7a6f-4592-8b4c-bea69d545756 2024-03-25T23:56:15.204 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6bee79ee-7a6f-4592-8b4c-bea69d545756 2024-03-25T23:56:15.204 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 6bee79ee-7a6f-4592-8b4c-bea69d545756' 2024-03-25T23:56:15.205 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:56:15.224 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAfDwJmnKGLDRAAOmfUyHGDbnJko0u22V6BoQ== 2024-03-25T23:56:15.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAfDwJmnKGLDRAAOmfUyHGDbnJko0u22V6BoQ=="}' 2024-03-25T23:56:15.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6bee79ee-7a6f-4592-8b4c-bea69d545756 -i td/osd-scrub-repair/7/new.json 2024-03-25T23:56:15.580 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:56:15.597 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/7/new.json 2024-03-25T23:56:15.600 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAfDwJmnKGLDRAAOmfUyHGDbnJko0u22V6BoQ== --osd-uuid 6bee79ee-7a6f-4592-8b4c-bea69d545756 2024-03-25T23:56:15.630 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:56:15.632+0000 7fe6c76b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:56:15.709 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:56:15.708+0000 7fe6c76b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:56:15.710 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:56:15.712+0000 7fe6c76b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:56:15.711 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:56:15.712+0000 7fe6c76b27c0 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-03-25T23:56:15.711 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:56:15.712+0000 7fe6c76b27c0 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-03-25T23:56:15.711 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:56:15.712+0000 7fe6c76b27c0 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-03-25T23:56:15.823 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:56:15.824+0000 7fe6c76b27c0 -1 bluestore(td/osd-scrub-repair/7) _read_fsid unparsable uuid 2024-03-25T23:56:23.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/7/keyring 2024-03-25T23:56:23.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:56:23.169 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd7 key to auth repository 2024-03-25T23:56:23.170 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-03-25T23:56:23.170 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:56:23.552 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.7 2024-03-25T23:56:23.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-03-25T23:56:23.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:56:23.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:56:23.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:56:23.554 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:56:23.585 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:56:23.584+0000 7fb9dd97c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:56:23.587 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:56:23.588+0000 7fb9dd97c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:56:23.588 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:56:23.588+0000 7fb9dd97c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:56:23.895 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-03-25T23:56:23.895 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:56:23.895 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-03-25T23:56:23.895 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:56:23.895 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:56:23.895 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:56:23.896 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:56:23.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:56:23.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:56:23.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:56:24.153 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:56:24.152+0000 7fb9dd97c7c0 -1 Falling back to public interface 2024-03-25T23:56:24.196 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:56:25.198 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:56:25.198 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:56:25.198 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:56:25.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:56:25.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:56:25.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:56:25.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:56:26.514 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:56:26.514 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:56:26.515 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:56:26.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:56:26.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:56:26.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:56:26.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:56:26.927 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:56:26.927+0000 7fb9dd97c7c0 -1 osd.7 0 log_to_monitors true 2024-03-25T23:56:27.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:56:27.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:56:27.840 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:56:27.841 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:56:27.841 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:56:27.841 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:56:28.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:56:29.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:56:29.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:56:29.343 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:56:29.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:56:29.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:56:29.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:56:29.701 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:56:30.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:56:30.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:56:30.703 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:56:30.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:56:30.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:56:30.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:56:31.154 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:56:32.156 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:56:32.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:56:32.157 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:56:32.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:56:32.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:56:32.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:56:32.479 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:56:33.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:56:33.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:56:33.481 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:56:33.482 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:56:33.482 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:56:33.482 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:56:34.011 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:56:35.013 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:56:35.013 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:56:35.013 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:56:35.014 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:56:35.014 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:56:35.014 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:56:35.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:56:36.320 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:56:36.321 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:56:36.321 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:56:36.321 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:56:36.321 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:56:36.321 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:56:36.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:56:37.647 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:56:37.647 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:56:37.648 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:56:37.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:56:37.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:56:37.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:56:38.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:56:39.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:56:39.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:56:39.200 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:56:39.200 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:56:39.200 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:56:39.200 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:56:39.520 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:56:40.522 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:56:40.522 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:56:40.522 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:56:40.523 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:56:40.523 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:56:40.523 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:56:40.905 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:56:41.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:56:41.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:56:41.907 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:56:41.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:56:41.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:56:41.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:56:42.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:56:43.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:56:43.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:56:43.779 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:56:43.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:56:43.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:56:43.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:56:44.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:56:45.610 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:56:45.610 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:56:45.610 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:56:45.611 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:56:45.611 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:56:45.611 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:56:46.539 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:56:47.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:56:47.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:56:47.542 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:56:47.542 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:56:47.542 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:56:47.542 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:56:48.514 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:56:49.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:56:49.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:56:49.516 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:56:49.516 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:56:49.516 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:56:49.516 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:56:50.300 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:56:51.302 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:56:51.302 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:56:51.303 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:56:51.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:56:51.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:56:51.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:56:51.841 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:56:52.843 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:56:52.843 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:56:52.843 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:56:52.844 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:56:52.844 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:56:52.844 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:56:53.318 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:56:54.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:56:54.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:56:54.319 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-25T23:56:54.320 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-25T23:56:54.320 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:56:54.320 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T23:56:54.728 INFO:tasks.workunit.client.0.smithi129.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/2907968637,v1:127.0.0.1:6859/2907968637] [v2:127.0.0.1:6860/2907968637,v1:127.0.0.1:6861/2907968637] exists,up 6bee79ee-7a6f-4592-8b4c-bea69d545756 2024-03-25T23:56:54.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:56:54.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:56:54.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:56:54.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-25T23:56:54.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 8 2024-03-25T23:56:54.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:56:54.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:56:54.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2024-03-25T23:56:54.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:56:54.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/8 2024-03-25T23:56:54.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:56:54.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:56:54.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:56:54.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:56:54.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2024-03-25T23:56:54.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2024-03-25T23:56:54.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:56:54.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:56:54.733 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:56:54.733 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:56:54.733 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:56:54.733 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:56:54.733 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:56:54.733 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:56:54.733 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:56:54.733 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:56:54.733 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:56:54.733 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:56:54.733 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:56:54.733 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:56:54.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:56:54.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:56:54.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:56:54.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:56:54.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:56:54.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:56:54.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:56:54.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T23:56:54.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/8 2024-03-25T23:56:54.735 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:56:54.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0c895480-c03d-4896-8663-5b0e77eea4cb 2024-03-25T23:56:54.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 0c895480-c03d-4896-8663-5b0e77eea4cb' 2024-03-25T23:56:54.735 INFO:tasks.workunit.client.0.smithi129.stdout:add osd8 0c895480-c03d-4896-8663-5b0e77eea4cb 2024-03-25T23:56:54.735 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:56:54.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBGDwJmDH5HLRAAzHJpbHdciNeVSlQetEgIxg== 2024-03-25T23:56:54.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBGDwJmDH5HLRAAzHJpbHdciNeVSlQetEgIxg=="}' 2024-03-25T23:56:54.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0c895480-c03d-4896-8663-5b0e77eea4cb -i td/osd-scrub-repair/8/new.json 2024-03-25T23:56:55.164 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:56:55.183 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/8/new.json 2024-03-25T23:56:55.184 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBGDwJmDH5HLRAAzHJpbHdciNeVSlQetEgIxg== --osd-uuid 0c895480-c03d-4896-8663-5b0e77eea4cb 2024-03-25T23:56:55.220 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:56:55.219+0000 7f8771fa77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:56:55.264 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:56:55.263+0000 7f8771fa77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:56:55.266 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:56:55.267+0000 7f8771fa77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:56:55.266 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:56:55.267+0000 7f8771fa77c0 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-03-25T23:56:55.266 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:56:55.267+0000 7f8771fa77c0 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-03-25T23:56:55.266 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:56:55.267+0000 7f8771fa77c0 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-03-25T23:56:55.346 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:56:55.343+0000 7f8771fa77c0 -1 bluestore(td/osd-scrub-repair/8) _read_fsid unparsable uuid 2024-03-25T23:57:02.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/8/keyring 2024-03-25T23:57:02.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:57:02.569 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd8 key to auth repository 2024-03-25T23:57:02.569 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2024-03-25T23:57:02.569 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:57:02.948 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.8 2024-03-25T23:57:02.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2024-03-25T23:57:02.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:57:02.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:57:02.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:57:02.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:57:02.981 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:57:02.979+0000 7f4105b617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:57:03.013 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:57:03.011+0000 7f4105b617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:57:03.014 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:57:03.015+0000 7f4105b617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:57:03.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2024-03-25T23:57:03.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:57:03.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2024-03-25T23:57:03.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:57:03.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:57:03.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:57:03.285 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:57:03.285 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:57:03.285 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:57:03.285 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:57:03.572 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:57:03.571+0000 7f4105b617c0 -1 Falling back to public interface 2024-03-25T23:57:03.600 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:57:04.602 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:57:04.602 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:57:04.602 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:57:04.603 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:57:04.603 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:57:04.603 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:57:04.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:57:05.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:57:05.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:57:05.922 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:57:05.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:57:05.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:57:05.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:57:06.232 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:57:06.231+0000 7f4105b617c0 -1 osd.8 0 log_to_monitors true 2024-03-25T23:57:06.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:57:07.242 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:57:07.242 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:57:07.242 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:57:07.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:57:07.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:57:07.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:57:07.641 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:57:08.643 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:57:08.643 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:57:08.644 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:57:08.644 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:57:08.644 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:57:08.644 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:57:09.054 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:57:10.056 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:57:10.056 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:57:10.056 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:57:10.056 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:57:10.057 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:57:10.057 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:57:10.501 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:57:11.503 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:57:11.503 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:57:11.503 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:57:11.504 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:57:11.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:57:11.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:57:11.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:57:12.821 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:57:12.821 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:57:12.822 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:57:12.822 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:57:12.822 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:57:12.822 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:57:13.209 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:57:14.210 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:57:14.211 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:57:14.211 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:57:14.211 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:57:14.211 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:57:14.211 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:57:14.657 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:57:15.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:57:15.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:57:15.660 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:57:15.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:57:15.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:57:15.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:57:15.978 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:57:16.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:57:16.980 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:57:16.980 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:57:16.980 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:57:16.980 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:57:16.980 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:57:17.360 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:57:18.362 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:57:18.362 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:57:18.362 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:57:18.362 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:57:18.363 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:57:18.363 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:57:18.724 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:57:19.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:57:19.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:57:19.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:57:19.727 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:57:19.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:57:19.728 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:57:20.087 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:57:21.089 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:57:21.089 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:57:21.089 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:57:21.089 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:57:21.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:57:21.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:57:21.535 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:57:22.537 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:57:22.537 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:57:22.537 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:57:22.538 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:57:22.538 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:57:22.538 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:57:22.857 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:57:23.858 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:57:23.859 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:57:23.859 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:57:23.859 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:57:23.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:57:23.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:57:24.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:57:25.965 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:57:25.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:57:25.966 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:57:25.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:57:25.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:57:25.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:57:26.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:57:27.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:57:27.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:57:27.836 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:57:27.837 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:57:27.837 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:57:27.837 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:57:28.366 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:57:29.368 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:57:29.368 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:57:29.368 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:57:29.369 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:57:29.369 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:57:29.369 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:57:30.068 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:57:31.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:57:31.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:57:31.070 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:57:31.071 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:57:31.071 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:57:31.071 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:57:31.885 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:57:32.886 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:57:32.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:57:32.887 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-25T23:57:32.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-25T23:57:32.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:57:32.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:57:33.442 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:57:34.445 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:57:34.445 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:57:34.445 INFO:tasks.workunit.client.0.smithi129.stdout:21 2024-03-25T23:57:34.446 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-25T23:57:34.446 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:57:34.446 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T23:57:34.800 INFO:tasks.workunit.client.0.smithi129.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/2074416797,v1:127.0.0.1:6867/2074416797] [v2:127.0.0.1:6868/2074416797,v1:127.0.0.1:6869/2074416797] exists,up 0c895480-c03d-4896-8663-5b0e77eea4cb 2024-03-25T23:57:34.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:57:34.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:57:34.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:57:34.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-25T23:57:34.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 9 2024-03-25T23:57:34.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-25T23:57:34.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T23:57:34.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=9 2024-03-25T23:57:34.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T23:57:34.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/9 2024-03-25T23:57:34.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:57:34.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:57:34.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:57:34.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:57:34.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-25T23:57:34.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-25T23:57:34.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T23:57:34.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T23:57:34.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:57:34.804 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T23:57:34.804 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:57:34.804 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:57:34.804 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:57:34.804 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:57:34.804 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:57:34.804 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:57:34.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:57:34.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:57:34.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T23:57:34.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T23:57:34.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:57:34.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:57:34.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:57:34.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:57:34.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:57:34.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:57:34.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T23:57:34.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T23:57:34.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/9 2024-03-25T23:57:34.806 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T23:57:34.807 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5fb70f5f-d7d5-45c0-b9fd-86c8b6646430 2024-03-25T23:57:34.807 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd9 5fb70f5f-d7d5-45c0-b9fd-86c8b6646430' 2024-03-25T23:57:34.807 INFO:tasks.workunit.client.0.smithi129.stdout:add osd9 5fb70f5f-d7d5-45c0-b9fd-86c8b6646430 2024-03-25T23:57:34.807 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T23:57:34.828 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBuDwJmWNKDMRAAPakFvvAw6vjFe1DtUU/5wg== 2024-03-25T23:57:34.828 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBuDwJmWNKDMRAAPakFvvAw6vjFe1DtUU/5wg=="}' 2024-03-25T23:57:34.828 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5fb70f5f-d7d5-45c0-b9fd-86c8b6646430 -i td/osd-scrub-repair/9/new.json 2024-03-25T23:57:35.245 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:57:35.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/9/new.json 2024-03-25T23:57:35.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 9 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBuDwJmWNKDMRAAPakFvvAw6vjFe1DtUU/5wg== --osd-uuid 5fb70f5f-d7d5-45c0-b9fd-86c8b6646430 2024-03-25T23:57:35.292 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:57:35.290+0000 7f527bb567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:57:35.347 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:57:35.346+0000 7f527bb567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:57:35.348 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:57:35.346+0000 7f527bb567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:57:35.349 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:57:35.346+0000 7f527bb567c0 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-03-25T23:57:35.349 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:57:35.346+0000 7f527bb567c0 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-03-25T23:57:35.349 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:57:35.346+0000 7f527bb567c0 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-03-25T23:57:35.432 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:57:35.434+0000 7f527bb567c0 -1 bluestore(td/osd-scrub-repair/9) _read_fsid unparsable uuid 2024-03-25T23:57:42.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/9/keyring 2024-03-25T23:57:42.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T23:57:42.486 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd9 key to auth repository 2024-03-25T23:57:42.486 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd9 key to auth repository 2024-03-25T23:57:42.487 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T23:57:42.878 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.9 2024-03-25T23:57:42.879 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.9 2024-03-25T23:57:42.879 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 9 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:57:42.880 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T23:57:42.880 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T23:57:42.880 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T23:57:42.910 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:57:42.910+0000 7f051f9dd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:57:42.912 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:57:42.910+0000 7f051f9dd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:57:42.913 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:57:42.910+0000 7f051f9dd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:57:43.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 9 2024-03-25T23:57:43.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:57:43.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-25T23:57:43.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:57:43.226 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:57:43.226 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:57:43.226 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-25T23:57:43.226 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:57:43.226 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:57:43.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:57:43.477 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:57:43.474+0000 7f051f9dd7c0 -1 Falling back to public interface 2024-03-25T23:57:43.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:57:44.566 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:57:44.566 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:57:44.566 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-25T23:57:44.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:57:44.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:57:44.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:57:44.889 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:57:45.891 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:57:45.891 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:57:45.891 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-25T23:57:45.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:57:45.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:57:45.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:57:46.196 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:57:46.194+0000 7f051f9dd7c0 -1 osd.9 0 log_to_monitors true 2024-03-25T23:57:46.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:57:47.209 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:57:47.209 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:57:47.210 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-25T23:57:47.210 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:57:47.210 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:57:47.210 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:57:47.673 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:57:48.675 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:57:48.675 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:57:48.676 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-25T23:57:48.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:57:48.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:57:48.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:57:49.100 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:57:50.101 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:57:50.101 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:57:50.101 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-25T23:57:50.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T23:57:50.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:57:50.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:57:50.606 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:57:51.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:57:51.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:57:51.608 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-25T23:57:51.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T23:57:51.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:57:51.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:57:51.931 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:57:52.932 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:57:52.932 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:57:52.933 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-25T23:57:52.933 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T23:57:52.933 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:57:52.933 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:57:53.285 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:57:54.287 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:57:54.287 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:57:54.287 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-25T23:57:54.288 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T23:57:54.288 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:57:54.288 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:57:54.738 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:57:55.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:57:55.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:57:55.741 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-25T23:57:55.741 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T23:57:55.741 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:57:55.741 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:57:56.121 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:57:57.122 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:57:57.122 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:57:57.123 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-25T23:57:57.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T23:57:57.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:57:57.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:57:57.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:57:58.454 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:57:58.455 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:57:58.455 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-25T23:57:58.455 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T23:57:58.455 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:57:58.455 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:57:58.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:57:59.909 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:57:59.909 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:57:59.910 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-25T23:57:59.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T23:57:59.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:57:59.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:58:00.327 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:58:01.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:58:01.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:58:01.329 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-25T23:58:01.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T23:58:01.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:58:01.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:58:01.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:58:02.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:58:02.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:58:02.650 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-25T23:58:02.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T23:58:02.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:58:02.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:58:03.275 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:58:04.277 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:58:04.277 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:58:04.277 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-25T23:58:04.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T23:58:04.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:58:04.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:58:04.765 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:58:05.766 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:58:05.766 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:58:05.767 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-25T23:58:05.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T23:58:05.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:58:05.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:58:06.854 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:58:07.852 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:58:07.852 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:58:07.852 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-25T23:58:07.853 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T23:58:07.853 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:58:07.853 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:58:08.367 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:58:09.368 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:58:09.368 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:58:09.369 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-25T23:58:09.369 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T23:58:09.369 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:58:09.369 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:58:09.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:58:10.871 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:58:10.871 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:58:10.871 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-25T23:58:10.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T23:58:10.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:58:10.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:58:11.400 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:58:12.401 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:58:12.401 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:58:12.402 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-25T23:58:12.402 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-25T23:58:12.402 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:58:12.402 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:58:13.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:58:14.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:58:14.174 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:58:14.174 INFO:tasks.workunit.client.0.smithi129.stdout:21 2024-03-25T23:58:14.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-25T23:58:14.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:58:14.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:58:14.678 INFO:tasks.workunit.client.0.smithi129.stdout:osd.9 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/2946427544,v1:127.0.0.1:6875/2946427544] [v2:127.0.0.1:6876/2946427544,v1:127.0.0.1:6877/2946427544] exists,up 5fb70f5f-d7d5-45c0-b9fd-86c8b6646430 2024-03-25T23:58:14.679 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:58:14.679 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:58:14.679 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:58:14.679 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: create_rbd_pool 2024-03-25T23:58:14.679 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-25T23:58:14.987 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'rbd' does not exist 2024-03-25T23:58:15.004 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-25T23:58:15.004 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-25T23:58:15.473 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'rbd' created 2024-03-25T23:58:15.495 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T23:58:16.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-25T23:58:20.104 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:835: corrupt_and_repair_lrc: wait_for_clean 2024-03-25T23:58:20.104 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:58:20.104 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:58:20.104 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:58:20.105 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:58:20.105 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:58:20.106 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:58:20.106 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:58:20.106 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:58:20.106 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:58:20.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:58:20.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:58:20.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:58:20.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:58:20.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:58:20.238 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:58:20.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:58:20.778 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:58:20.779 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:58:20.779 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:58:20.779 INFO:tasks.workunit.client.0.smithi129.stderr:4 2024-03-25T23:58:20.779 INFO:tasks.workunit.client.0.smithi129.stderr:5 2024-03-25T23:58:20.779 INFO:tasks.workunit.client.0.smithi129.stderr:6 2024-03-25T23:58:20.779 INFO:tasks.workunit.client.0.smithi129.stderr:7 2024-03-25T23:58:20.779 INFO:tasks.workunit.client.0.smithi129.stderr:8 2024-03-25T23:58:20.779 INFO:tasks.workunit.client.0.smithi129.stderr:9' 2024-03-25T23:58:20.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:58:20.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:58:20.779 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:58:20.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836562 2024-03-25T23:58:20.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836562 2024-03-25T23:58:20.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562' 2024-03-25T23:58:20.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:58:20.927 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:58:21.109 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673031 2024-03-25T23:58:21.109 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673031 2024-03-25T23:58:21.109 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673031' 2024-03-25T23:58:21.109 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:58:21.109 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:58:21.251 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509502 2024-03-25T23:58:21.251 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509502 2024-03-25T23:58:21.251 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673031 2-64424509502' 2024-03-25T23:58:21.252 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:58:21.252 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:58:21.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345975 2024-03-25T23:58:21.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345975 2024-03-25T23:58:21.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673031 2-64424509502 3-85899345975' 2024-03-25T23:58:21.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:58:21.392 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T23:58:21.535 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182448 2024-03-25T23:58:21.536 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182448 2024-03-25T23:58:21.536 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673031 2-64424509502 3-85899345975 4-107374182448' 2024-03-25T23:58:21.536 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:58:21.536 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T23:58:21.684 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018922 2024-03-25T23:58:21.684 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018922 2024-03-25T23:58:21.684 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673031 2-64424509502 3-85899345975 4-107374182448 5-128849018922' 2024-03-25T23:58:21.684 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:58:21.684 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T23:58:21.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855390 2024-03-25T23:58:21.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855390 2024-03-25T23:58:21.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673031 2-64424509502 3-85899345975 4-107374182448 5-128849018922 6-150323855390' 2024-03-25T23:58:21.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:58:21.830 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-25T23:58:21.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691868 2024-03-25T23:58:21.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691868 2024-03-25T23:58:21.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673031 2-64424509502 3-85899345975 4-107374182448 5-128849018922 6-150323855390 7-171798691868' 2024-03-25T23:58:21.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:58:21.973 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-25T23:58:22.115 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528338 2024-03-25T23:58:22.115 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528338 2024-03-25T23:58:22.115 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673031 2-64424509502 3-85899345975 4-107374182448 5-128849018922 6-150323855390 7-171798691868 8-193273528338' 2024-03-25T23:58:22.115 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:58:22.115 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-25T23:58:22.255 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364807 2024-03-25T23:58:22.255 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364807 2024-03-25T23:58:22.255 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673031 2-64424509502 3-85899345975 4-107374182448 5-128849018922 6-150323855390 7-171798691868 8-193273528338 9-214748364807' 2024-03-25T23:58:22.255 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:58:22.256 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836562 2024-03-25T23:58:22.256 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:58:22.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:58:22.257 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836562 2024-03-25T23:58:22.257 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:58:22.258 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836562 2024-03-25T23:58:22.258 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836562' 2024-03-25T23:58:22.258 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 21474836562 2024-03-25T23:58:22.259 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:58:22.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836562 -lt 21474836562 2024-03-25T23:58:22.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:58:22.563 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673031 2024-03-25T23:58:22.563 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:58:22.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:58:22.566 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673031 2024-03-25T23:58:22.566 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:58:22.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673031 2024-03-25T23:58:22.568 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 42949673031 2024-03-25T23:58:22.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673031' 2024-03-25T23:58:22.568 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:58:22.885 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673031 -lt 42949673031 2024-03-25T23:58:22.886 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:58:22.886 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509502 2024-03-25T23:58:22.886 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:58:22.889 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:58:22.890 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509502 2024-03-25T23:58:22.890 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:58:22.891 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509502 2024-03-25T23:58:22.891 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509502' 2024-03-25T23:58:22.891 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 64424509502 2024-03-25T23:58:22.892 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:58:23.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509502 -lt 64424509502 2024-03-25T23:58:23.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:58:23.232 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:58:23.233 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345975 2024-03-25T23:58:23.234 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:58:23.235 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345975 2024-03-25T23:58:23.235 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:58:23.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345975 2024-03-25T23:58:23.237 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.3 seq 85899345975 2024-03-25T23:58:23.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345975' 2024-03-25T23:58:23.237 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:58:23.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345976 -lt 85899345975 2024-03-25T23:58:23.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:58:23.568 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182448 2024-03-25T23:58:23.568 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:58:23.569 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T23:58:23.570 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182448 2024-03-25T23:58:23.570 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:58:23.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182448 2024-03-25T23:58:23.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182448' 2024-03-25T23:58:23.572 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.4 seq 107374182448 2024-03-25T23:58:23.572 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T23:58:23.903 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182448 -lt 107374182448 2024-03-25T23:58:23.903 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:58:23.904 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018922 2024-03-25T23:58:23.904 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:58:23.905 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T23:58:23.906 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018922 2024-03-25T23:58:23.906 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:58:23.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018922 2024-03-25T23:58:23.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018922' 2024-03-25T23:58:23.908 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.5 seq 128849018922 2024-03-25T23:58:23.908 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T23:58:24.247 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018923 -lt 128849018922 2024-03-25T23:58:24.247 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:58:24.248 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855390 2024-03-25T23:58:24.248 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:58:24.250 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-25T23:58:24.250 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855390 2024-03-25T23:58:24.250 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:58:24.253 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855390 2024-03-25T23:58:24.254 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.6 seq 150323855390 2024-03-25T23:58:24.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855390' 2024-03-25T23:58:24.254 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-25T23:58:24.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855390 -lt 150323855390 2024-03-25T23:58:24.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:58:24.588 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691868 2024-03-25T23:58:24.588 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:58:24.590 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-25T23:58:24.591 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691868 2024-03-25T23:58:24.591 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:58:24.593 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691868 2024-03-25T23:58:24.594 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.7 seq 171798691868 2024-03-25T23:58:24.594 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691868' 2024-03-25T23:58:24.594 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-25T23:58:24.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691869 -lt 171798691868 2024-03-25T23:58:24.921 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:58:24.921 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:58:24.921 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528338 2024-03-25T23:58:24.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-25T23:58:24.923 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528338 2024-03-25T23:58:24.924 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:58:24.925 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528338 2024-03-25T23:58:24.925 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528338' 2024-03-25T23:58:24.925 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.8 seq 193273528338 2024-03-25T23:58:24.925 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-25T23:58:25.273 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528338 -lt 193273528338 2024-03-25T23:58:25.273 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:58:25.274 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364807 2024-03-25T23:58:25.274 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:58:25.275 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-25T23:58:25.276 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364807 2024-03-25T23:58:25.276 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:58:25.277 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364807 2024-03-25T23:58:25.277 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.9 seq 214748364807 2024-03-25T23:58:25.277 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364807' 2024-03-25T23:58:25.277 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-25T23:58:25.590 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364807 -lt 214748364807 2024-03-25T23:58:25.590 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:58:25.591 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:58:25.591 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:58:25.968 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-25T23:58:25.968 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:58:25.969 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:58:25.969 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:58:25.969 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:58:25.969 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:58:25.969 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:58:25.970 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:58:26.277 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T23:58:26.277 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:58:26.278 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:58:26.278 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:58:26.673 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-25T23:58:26.673 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:58:26.673 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:58:26.674 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: create_ec_pool ecpool true k=4 m=2 l=3 plugin=lrc 2024-03-25T23:58:26.674 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-25T23:58:26.674 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-25T23:58:26.674 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-03-25T23:58:26.674 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-25T23:58:26.674 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2024-03-25T23:58:27.171 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-25T23:58:27.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-25T23:58:28.098 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'ecpool' created 2024-03-25T23:58:28.120 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T23:58:29.122 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-03-25T23:58:29.122 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-03-25T23:58:29.776 INFO:tasks.workunit.client.0.smithi129.stderr:set pool 2 allow_ec_overwrites to true 2024-03-25T23:58:29.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-25T23:58:29.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:58:29.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:58:29.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:58:29.798 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:58:29.798 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:58:29.798 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:58:29.799 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:58:29.799 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:58:29.799 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:58:29.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:58:29.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:58:29.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:58:29.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:58:29.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:58:29.898 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:58:30.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:58:30.308 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:58:30.308 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:58:30.308 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:58:30.308 INFO:tasks.workunit.client.0.smithi129.stderr:4 2024-03-25T23:58:30.308 INFO:tasks.workunit.client.0.smithi129.stderr:5 2024-03-25T23:58:30.308 INFO:tasks.workunit.client.0.smithi129.stderr:6 2024-03-25T23:58:30.308 INFO:tasks.workunit.client.0.smithi129.stderr:7 2024-03-25T23:58:30.308 INFO:tasks.workunit.client.0.smithi129.stderr:8 2024-03-25T23:58:30.308 INFO:tasks.workunit.client.0.smithi129.stderr:9' 2024-03-25T23:58:30.308 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:58:30.308 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:58:30.309 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:58:30.544 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836565 2024-03-25T23:58:30.544 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836565 2024-03-25T23:58:30.544 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565' 2024-03-25T23:58:30.544 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:58:30.545 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:58:30.725 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673033 2024-03-25T23:58:30.726 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673033 2024-03-25T23:58:30.726 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673033' 2024-03-25T23:58:30.726 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:58:30.726 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:58:30.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509505 2024-03-25T23:58:30.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509505 2024-03-25T23:58:30.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673033 2-64424509505' 2024-03-25T23:58:30.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:58:30.869 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:58:31.009 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345978 2024-03-25T23:58:31.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345978 2024-03-25T23:58:31.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673033 2-64424509505 3-85899345978' 2024-03-25T23:58:31.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:58:31.010 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T23:58:31.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182451 2024-03-25T23:58:31.143 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182451 2024-03-25T23:58:31.143 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673033 2-64424509505 3-85899345978 4-107374182451' 2024-03-25T23:58:31.143 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:58:31.143 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T23:58:31.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018925 2024-03-25T23:58:31.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018925 2024-03-25T23:58:31.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673033 2-64424509505 3-85899345978 4-107374182451 5-128849018925' 2024-03-25T23:58:31.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:58:31.278 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T23:58:31.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855393 2024-03-25T23:58:31.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855393 2024-03-25T23:58:31.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673033 2-64424509505 3-85899345978 4-107374182451 5-128849018925 6-150323855393' 2024-03-25T23:58:31.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:58:31.419 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-25T23:58:31.558 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691871 2024-03-25T23:58:31.558 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691871 2024-03-25T23:58:31.558 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673033 2-64424509505 3-85899345978 4-107374182451 5-128849018925 6-150323855393 7-171798691871' 2024-03-25T23:58:31.558 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:58:31.558 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-25T23:58:31.690 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528341 2024-03-25T23:58:31.690 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528341 2024-03-25T23:58:31.690 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673033 2-64424509505 3-85899345978 4-107374182451 5-128849018925 6-150323855393 7-171798691871 8-193273528341' 2024-03-25T23:58:31.690 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:58:31.690 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-25T23:58:31.822 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364810 2024-03-25T23:58:31.822 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364810 2024-03-25T23:58:31.822 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673033 2-64424509505 3-85899345978 4-107374182451 5-128849018925 6-150323855393 7-171798691871 8-193273528341 9-214748364810' 2024-03-25T23:58:31.822 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:58:31.822 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836565 2024-03-25T23:58:31.822 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:58:31.823 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:58:31.823 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836565 2024-03-25T23:58:31.824 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:58:31.824 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836565 2024-03-25T23:58:31.825 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 21474836565 2024-03-25T23:58:31.825 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836565' 2024-03-25T23:58:31.825 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:58:32.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836565 -lt 21474836565 2024-03-25T23:58:32.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:58:32.120 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673033 2024-03-25T23:58:32.120 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:58:32.122 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:58:32.123 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673033 2024-03-25T23:58:32.123 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:58:32.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673033 2024-03-25T23:58:32.125 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 42949673033 2024-03-25T23:58:32.125 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673033' 2024-03-25T23:58:32.125 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:58:32.514 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673034 -lt 42949673033 2024-03-25T23:58:32.514 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:58:32.515 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509505 2024-03-25T23:58:32.515 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:58:32.516 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:58:32.517 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509505 2024-03-25T23:58:32.517 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:58:32.519 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509505 2024-03-25T23:58:32.519 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 64424509505 2024-03-25T23:58:32.519 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509505' 2024-03-25T23:58:32.519 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:58:32.842 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509505 -lt 64424509505 2024-03-25T23:58:32.842 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:58:32.842 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345978 2024-03-25T23:58:32.842 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:58:32.843 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:58:32.844 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345978 2024-03-25T23:58:32.844 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:58:32.845 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345978 2024-03-25T23:58:32.845 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.3 seq 85899345978 2024-03-25T23:58:32.845 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345978' 2024-03-25T23:58:32.845 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:58:33.155 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345979 -lt 85899345978 2024-03-25T23:58:33.155 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:58:33.155 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182451 2024-03-25T23:58:33.156 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:58:33.156 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T23:58:33.157 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182451 2024-03-25T23:58:33.158 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:58:33.159 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182451 2024-03-25T23:58:33.159 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.4 seq 107374182451 2024-03-25T23:58:33.159 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182451' 2024-03-25T23:58:33.159 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T23:58:33.479 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182451 -lt 107374182451 2024-03-25T23:58:33.480 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:58:33.480 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018925 2024-03-25T23:58:33.480 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:58:33.482 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T23:58:33.483 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018925 2024-03-25T23:58:33.483 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:58:33.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018925 2024-03-25T23:58:33.485 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.5 seq 128849018925 2024-03-25T23:58:33.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018925' 2024-03-25T23:58:33.485 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T23:58:33.812 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018925 -lt 128849018925 2024-03-25T23:58:33.812 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:58:33.813 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855393 2024-03-25T23:58:33.813 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:58:33.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-25T23:58:33.815 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855393 2024-03-25T23:58:33.815 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:58:33.816 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855393 2024-03-25T23:58:33.816 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.6 seq 150323855393 2024-03-25T23:58:33.816 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855393' 2024-03-25T23:58:33.817 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-25T23:58:34.129 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855393 -lt 150323855393 2024-03-25T23:58:34.130 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:58:34.130 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691871 2024-03-25T23:58:34.130 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:58:34.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-25T23:58:34.133 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691871 2024-03-25T23:58:34.133 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:58:34.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691871 2024-03-25T23:58:34.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691871' 2024-03-25T23:58:34.135 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.7 seq 171798691871 2024-03-25T23:58:34.135 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-25T23:58:34.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691872 -lt 171798691871 2024-03-25T23:58:34.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:58:34.484 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:58:34.485 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528341 2024-03-25T23:58:34.486 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-25T23:58:34.490 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528341 2024-03-25T23:58:34.490 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:58:34.490 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528341 2024-03-25T23:58:34.490 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528341' 2024-03-25T23:58:34.490 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.8 seq 193273528341 2024-03-25T23:58:34.490 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-25T23:58:34.865 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528341 -lt 193273528341 2024-03-25T23:58:34.866 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:58:34.866 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364810 2024-03-25T23:58:34.866 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:58:34.867 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-25T23:58:34.868 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364810 2024-03-25T23:58:34.868 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:58:34.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364810 2024-03-25T23:58:34.870 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.9 seq 214748364810 2024-03-25T23:58:34.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364810' 2024-03-25T23:58:34.870 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-25T23:58:35.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364810 -lt 214748364810 2024-03-25T23:58:35.198 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:58:35.198 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:58:35.198 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:58:35.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:58:35.551 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:58:35.551 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:58:35.551 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:58:35.551 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:58:35.551 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:58:35.551 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:58:35.551 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:58:35.906 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:58:35.907 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:58:35.907 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:58:35.907 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:58:36.251 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:58:36.251 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:58:36.251 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:58:36.251 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-25T23:58:36.251 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-03-25T23:58:36.251 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-03-25T23:58:36.252 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-03-25T23:58:36.252 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-03-25T23:58:36.252 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-25T23:58:36.252 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-25T23:58:36.252 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-03-25T23:58:36.252 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-25T23:58:36.252 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-25T23:58:36.252 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-25T23:58:36.788 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-25T23:58:36.809 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-25T23:58:37.444 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-25T23:58:37.466 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-25T23:58:37.466 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-25T23:58:37.466 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-25T23:58:38.122 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-03-25T23:58:38.122 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-03-25T23:58:38.122 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-25T23:58:38.122 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-03-25T23:58:38.122 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-25T23:58:38.601 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-03-25T23:58:38.602 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-03-25T23:58:38.602 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-25T23:58:38.602 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-03-25T23:58:38.602 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-03-25T23:58:38.603 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-03-25T23:58:38.603 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T23:58:38.944 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-25T23:58:38.945 INFO:tasks.workunit.client.0.smithi129.stderr:5 2024-03-25T23:58:38.945 INFO:tasks.workunit.client.0.smithi129.stderr:9 2024-03-25T23:58:38.945 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-25T23:58:38.945 INFO:tasks.workunit.client.0.smithi129.stderr:6 2024-03-25T23:58:38.945 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:58:38.945 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:58:38.945 INFO:tasks.workunit.client.0.smithi129.stderr:7' 2024-03-25T23:58:38.945 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 6 2 1 7 2024-03-25T23:58:38.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('5' '9' '0' '6' '2' '1' '7') 2024-03-25T23:58:38.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-03-25T23:58:38.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=5 2024-03-25T23:58:38.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=9 2024-03-25T23:58:38.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-03-25T23:58:38.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-25T23:58:38.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-03-25T23:58:38.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-03-25T23:58:38.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-25T23:58:38.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:58:38.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:58:38.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-25T23:58:38.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:58:38.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-03-25T23:58:38.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:58:38.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:58:38.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-25T23:58:38.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:58:38.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-25T23:58:38.949 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:58:38.949 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:58:38.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:58:38.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:58:38.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:58:39.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:58:39.258 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-03-25T23:58:39.258 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:58:39.258 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:58:39.258 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-25T23:58:39.258 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:58:39.258 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-25T23:58:39.258 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-03-25T23:58:41.594 INFO:tasks.workunit.client.0.smithi129.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-03-25T23:58:42.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-25T23:58:42.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:58:42.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:58:42.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-25T23:58:42.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:58:42.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-25T23:58:42.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:58:42.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:58:42.133 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:58:42.133 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:58:42.133 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-25T23:58:42.133 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-25T23:58:42.133 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:58:42.133 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:58:42.133 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:58:42.133 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:58:42.133 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:58:42.133 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:58:42.135 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:58:42.135 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:58:42.135 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:58:42.135 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:58:42.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:58:42.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:58:42.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:58:42.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:58:42.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:58:42.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:58:42.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:58:42.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:58:42.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:58:42.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:58:42.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-25T23:58:42.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-25T23:58:42.136 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.3 2024-03-25T23:58:42.137 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:58:42.137 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-25T23:58:42.137 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-25T23:58:42.137 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:58:42.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:58:42.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:58:42.169 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:58:42.169+0000 7fba6ecff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:58:42.172 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:58:42.173+0000 7fba6ecff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:58:42.174 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:58:42.177+0000 7fba6ecff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:58:42.483 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-25T23:58:42.483 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:58:42.483 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-25T23:58:42.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:58:42.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:58:42.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:58:42.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:58:42.484 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-25T23:58:42.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:58:42.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:58:42.700 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:58:42.701+0000 7fba6ecff7c0 -1 Falling back to public interface 2024-03-25T23:58:42.822 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:58:43.824 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:58:43.824 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:58:43.824 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:58:43.825 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:58:43.825 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:58:43.825 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:58:44.163 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:58:45.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:58:45.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:58:45.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:58:45.165 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:58:45.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:58:45.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:58:45.363 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:58:45.365+0000 7fba6ecff7c0 -1 osd.3 69 log_to_monitors true 2024-03-25T23:58:45.479 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:58:46.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:58:46.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:58:46.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:58:46.481 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:58:46.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:58:46.482 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T23:58:46.832 INFO:tasks.workunit.client.0.smithi129.stderr:osd.3 up in weight 1 up_from 73 up_thru 73 down_at 70 last_clean_interval [20,69) [v2:127.0.0.1:6826/3940177662,v1:127.0.0.1:6827/3940177662] [v2:127.0.0.1:6828/3940177662,v1:127.0.0.1:6829/3940177662] exists,up 25e4ca68-13d2-4df8-9930-c8c629bba9ef 2024-03-25T23:58:46.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:58:46.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:58:46.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:58:46.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:58:46.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:58:46.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:58:46.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:58:46.834 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:58:46.835 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:58:46.835 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:58:46.835 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:58:46.835 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:58:46.835 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:58:46.940 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:58:46.940 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:58:46.941 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:58:46.941 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:58:46.941 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:58:46.941 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:58:47.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:58:47.255 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:58:47.255 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:58:47.255 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:58:47.255 INFO:tasks.workunit.client.0.smithi129.stderr:4 2024-03-25T23:58:47.255 INFO:tasks.workunit.client.0.smithi129.stderr:5 2024-03-25T23:58:47.255 INFO:tasks.workunit.client.0.smithi129.stderr:6 2024-03-25T23:58:47.255 INFO:tasks.workunit.client.0.smithi129.stderr:7 2024-03-25T23:58:47.255 INFO:tasks.workunit.client.0.smithi129.stderr:8 2024-03-25T23:58:47.255 INFO:tasks.workunit.client.0.smithi129.stderr:9' 2024-03-25T23:58:47.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:58:47.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:58:47.256 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:58:47.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836569 2024-03-25T23:58:47.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836569 2024-03-25T23:58:47.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569' 2024-03-25T23:58:47.495 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:58:47.495 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:58:47.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673038 2024-03-25T23:58:47.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673038 2024-03-25T23:58:47.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673038' 2024-03-25T23:58:47.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:58:47.654 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:58:47.793 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509509 2024-03-25T23:58:47.793 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509509 2024-03-25T23:58:47.793 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673038 2-64424509509' 2024-03-25T23:58:47.793 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:58:47.793 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:58:47.932 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612610 2024-03-25T23:58:47.932 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612610 2024-03-25T23:58:47.932 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673038 2-64424509509 3-313532612610' 2024-03-25T23:58:47.932 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:58:47.932 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T23:58:48.064 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182455 2024-03-25T23:58:48.064 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182455 2024-03-25T23:58:48.064 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673038 2-64424509509 3-313532612610 4-107374182455' 2024-03-25T23:58:48.064 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:58:48.064 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T23:58:48.203 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018930 2024-03-25T23:58:48.203 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018930 2024-03-25T23:58:48.204 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673038 2-64424509509 3-313532612610 4-107374182455 5-128849018930' 2024-03-25T23:58:48.204 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:58:48.204 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T23:58:48.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855397 2024-03-25T23:58:48.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855397 2024-03-25T23:58:48.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673038 2-64424509509 3-313532612610 4-107374182455 5-128849018930 6-150323855397' 2024-03-25T23:58:48.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:58:48.345 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-25T23:58:48.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691875 2024-03-25T23:58:48.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691875 2024-03-25T23:58:48.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673038 2-64424509509 3-313532612610 4-107374182455 5-128849018930 6-150323855397 7-171798691875' 2024-03-25T23:58:48.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:58:48.489 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-25T23:58:48.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528345 2024-03-25T23:58:48.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528345 2024-03-25T23:58:48.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673038 2-64424509509 3-313532612610 4-107374182455 5-128849018930 6-150323855397 7-171798691875 8-193273528345' 2024-03-25T23:58:48.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:58:48.783 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-25T23:58:48.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364814 2024-03-25T23:58:48.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364814 2024-03-25T23:58:48.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673038 2-64424509509 3-313532612610 4-107374182455 5-128849018930 6-150323855397 7-171798691875 8-193273528345 9-214748364814' 2024-03-25T23:58:48.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:58:48.927 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836569 2024-03-25T23:58:48.928 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:58:48.928 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:58:48.929 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836569 2024-03-25T23:58:48.929 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:58:48.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836569 2024-03-25T23:58:48.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836569' 2024-03-25T23:58:48.930 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 21474836569 2024-03-25T23:58:48.930 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:58:49.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836569 -lt 21474836569 2024-03-25T23:58:49.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:58:49.228 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673038 2024-03-25T23:58:49.228 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:58:49.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:58:49.231 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673038 2024-03-25T23:58:49.231 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:58:49.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673038 2024-03-25T23:58:49.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673038' 2024-03-25T23:58:49.232 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 42949673038 2024-03-25T23:58:49.233 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:58:49.589 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673038 -lt 42949673038 2024-03-25T23:58:49.589 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:58:49.590 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509509 2024-03-25T23:58:49.590 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:58:49.591 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:58:49.592 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509509 2024-03-25T23:58:49.592 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:58:49.593 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509509 2024-03-25T23:58:49.593 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509509' 2024-03-25T23:58:49.594 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 64424509509 2024-03-25T23:58:49.594 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:58:49.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509509 -lt 64424509509 2024-03-25T23:58:49.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:58:49.920 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:58:49.920 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-313532612610 2024-03-25T23:58:49.921 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:58:49.922 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-313532612610 2024-03-25T23:58:49.922 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:58:49.924 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612610 2024-03-25T23:58:49.924 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 313532612610' 2024-03-25T23:58:49.924 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.3 seq 313532612610 2024-03-25T23:58:49.924 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:58:50.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612610 -lt 313532612610 2024-03-25T23:58:50.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:58:50.230 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182455 2024-03-25T23:58:50.231 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:58:50.233 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T23:58:50.234 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182455 2024-03-25T23:58:50.234 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:58:50.235 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182455 2024-03-25T23:58:50.235 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182455' 2024-03-25T23:58:50.235 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.4 seq 107374182455 2024-03-25T23:58:50.235 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T23:58:50.557 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182456 -lt 107374182455 2024-03-25T23:58:50.557 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:58:50.558 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018930 2024-03-25T23:58:50.558 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:58:50.559 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T23:58:50.560 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018930 2024-03-25T23:58:50.560 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:58:50.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018930 2024-03-25T23:58:50.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018930' 2024-03-25T23:58:50.562 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.5 seq 128849018930 2024-03-25T23:58:50.562 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T23:58:50.859 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018930 -lt 128849018930 2024-03-25T23:58:50.859 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:58:50.859 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855397 2024-03-25T23:58:50.860 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:58:50.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-25T23:58:50.862 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855397 2024-03-25T23:58:50.862 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:58:50.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855397 2024-03-25T23:58:50.864 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855397' 2024-03-25T23:58:50.864 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.6 seq 150323855397 2024-03-25T23:58:50.864 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-25T23:58:51.204 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855397 -lt 150323855397 2024-03-25T23:58:51.204 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:58:51.204 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691875 2024-03-25T23:58:51.205 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:58:51.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-25T23:58:51.208 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691875 2024-03-25T23:58:51.208 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:58:51.209 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691875 2024-03-25T23:58:51.209 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691875' 2024-03-25T23:58:51.209 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.7 seq 171798691875 2024-03-25T23:58:51.210 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-25T23:58:51.551 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691876 -lt 171798691875 2024-03-25T23:58:51.551 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:58:51.552 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528345 2024-03-25T23:58:51.552 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:58:51.554 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-25T23:58:51.555 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528345 2024-03-25T23:58:51.555 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:58:51.557 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528345 2024-03-25T23:58:51.557 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528345' 2024-03-25T23:58:51.557 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.8 seq 193273528345 2024-03-25T23:58:51.557 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-25T23:58:51.893 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528346 -lt 193273528345 2024-03-25T23:58:51.893 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:58:51.894 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364814 2024-03-25T23:58:51.894 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:58:51.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-25T23:58:51.897 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364814 2024-03-25T23:58:51.897 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:58:51.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364814 2024-03-25T23:58:51.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364814' 2024-03-25T23:58:51.899 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.9 seq 214748364814 2024-03-25T23:58:51.899 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-25T23:58:52.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364815 -lt 214748364814 2024-03-25T23:58:52.233 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:58:52.234 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:58:52.234 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:58:52.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:58:52.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:58:52.649 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:58:52.649 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:58:52.649 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:58:52.649 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:58:52.649 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:58:52.649 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:58:52.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:58:52.973 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:58:52.973 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:58:52.973 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:58:53.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:58:53.327 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:58:53.327 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:58:53.327 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-03-25T23:58:53.327 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-25T23:58:53.328 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-25T23:58:53.328 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-25T23:58:53.328 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-25T23:58:53.657 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-03-25T23:58:53.658 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-03-25T23:58:53.658 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-25T23:58:53.659 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-25T23:58:53.659 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:58:53.659 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:58:53.659 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:58:53.659 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:58:53.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-25T23:58:27.764477+0000 2024-03-25T23:58:53.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-25T23:58:54.266 INFO:tasks.workunit.client.0.smithi129.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-25T23:58:54.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-25T23:58:27.764477+0000 2024-03-25T23:58:54.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-25T23:58:54.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-25T23:58:27.764477+0000 2024-03-25T23:58:54.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-25T23:58:54.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-25T23:58:54.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:58:54.285 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:58:54.285 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:58:54.285 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:58:54.285 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:58:54.285 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:58:54.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:58:27.764477+0000 '>' 2024-03-25T23:58:27.764477+0000 2024-03-25T23:58:54.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:58:55.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:58:55.733 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:58:55.734 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:58:55.734 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:58:55.734 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:58:55.734 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:58:55.734 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:58:56.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:58:54.331151+0000 '>' 2024-03-25T23:58:27.764477+0000 2024-03-25T23:58:56.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-25T23:58:56.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-25T23:58:56.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:58:56.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:58:56.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-03-25T23:58:56.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:58:56.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-25T23:58:56.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:58:56.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:58:56.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-03-25T23:58:56.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:58:56.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-03-25T23:58:56.070 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:58:56.070 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:58:56.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:58:56.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:58:56.071 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:58:56.380 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:58:56.380 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-25T23:58:56.380 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:58:56.380 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:58:56.380 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-03-25T23:58:56.380 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:58:56.380 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-03-25T23:58:56.380 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-03-25T23:58:56.779 INFO:tasks.workunit.client.0.smithi129.stdout:_ 2024-03-25T23:58:56.779 INFO:tasks.workunit.client.0.smithi129.stdout:hinfo_key 2024-03-25T23:58:56.779 INFO:tasks.workunit.client.0.smithi129.stdout:snapset 2024-03-25T23:58:57.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-03-25T23:58:57.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:58:57.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:58:57.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-03-25T23:58:57.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:58:57.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-03-25T23:58:57.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:58:57.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:58:57.071 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:58:57.071 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:58:57.071 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-25T23:58:57.071 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-25T23:58:57.071 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:58:57.071 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:58:57.071 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:58:57.071 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:58:57.071 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:58:57.071 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:58:57.073 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:58:57.073 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:58:57.073 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:58:57.073 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:58:57.073 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:58:57.073 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:58:57.073 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:58:57.073 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:58:57.074 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:58:57.074 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:58:57.074 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:58:57.074 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:58:57.074 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:58:57.074 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:58:57.074 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-03-25T23:58:57.074 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-03-25T23:58:57.074 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.9 2024-03-25T23:58:57.075 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:58:57.075 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-03-25T23:58:57.075 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-03-25T23:58:57.075 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:58:57.076 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:58:57.076 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:58:57.113 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:58:57.112+0000 7f34c60727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:58:57.116 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:58:57.116+0000 7f34c60727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:58:57.117 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:58:57.116+0000 7f34c60727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:58:57.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-03-25T23:58:57.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:58:57.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-25T23:58:57.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:58:57.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:58:57.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:58:57.486 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:58:57.486 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-25T23:58:57.486 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:58:57.486 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:58:57.675 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:58:57.676+0000 7f34c60727c0 -1 Falling back to public interface 2024-03-25T23:58:57.895 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:58:58.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:58:58.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:58:58.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:58:58.897 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:58:58.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:58:58.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:58:59.214 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:59:00.215 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:59:00.216 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:59:00.216 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:59:00.216 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:59:00.216 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:59:00.216 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:59:00.539 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:59:01.165 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:59:01.164+0000 7f34c60727c0 -1 osd.9 74 log_to_monitors true 2024-03-25T23:59:01.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:59:01.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:59:01.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:59:01.541 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:59:01.542 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:59:01.542 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:59:01.871 INFO:tasks.workunit.client.0.smithi129.stderr:osd.9 up in weight 1 up_from 78 up_thru 0 down_at 75 last_clean_interval [50,74) [v2:127.0.0.1:6874/1070220194,v1:127.0.0.1:6875/1070220194] [v2:127.0.0.1:6876/1070220194,v1:127.0.0.1:6877/1070220194] exists,up 5fb70f5f-d7d5-45c0-b9fd-86c8b6646430 2024-03-25T23:59:01.871 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:59:01.871 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:59:01.871 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:59:01.871 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:59:01.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:59:01.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:59:01.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:59:01.872 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:59:01.872 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:59:01.873 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:59:01.873 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:59:01.873 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:59:01.873 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:59:01.971 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:59:01.971 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:59:01.971 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:59:01.971 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:59:01.971 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:59:01.971 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:59:02.443 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:59:02.443 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:59:02.443 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:59:02.443 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:59:02.443 INFO:tasks.workunit.client.0.smithi129.stderr:4 2024-03-25T23:59:02.443 INFO:tasks.workunit.client.0.smithi129.stderr:5 2024-03-25T23:59:02.443 INFO:tasks.workunit.client.0.smithi129.stderr:6 2024-03-25T23:59:02.443 INFO:tasks.workunit.client.0.smithi129.stderr:7 2024-03-25T23:59:02.443 INFO:tasks.workunit.client.0.smithi129.stderr:8 2024-03-25T23:59:02.443 INFO:tasks.workunit.client.0.smithi129.stderr:9' 2024-03-25T23:59:02.444 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:59:02.444 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:02.444 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:59:02.739 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836573 2024-03-25T23:59:02.739 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836573 2024-03-25T23:59:02.739 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573' 2024-03-25T23:59:02.739 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:02.739 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:59:02.905 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673042 2024-03-25T23:59:02.905 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673042 2024-03-25T23:59:02.905 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673042' 2024-03-25T23:59:02.906 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:02.906 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:59:03.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509513 2024-03-25T23:59:03.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509513 2024-03-25T23:59:03.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673042 2-64424509513' 2024-03-25T23:59:03.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:03.344 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:59:03.602 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612614 2024-03-25T23:59:03.602 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612614 2024-03-25T23:59:03.602 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673042 2-64424509513 3-313532612614' 2024-03-25T23:59:03.602 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:03.603 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T23:59:03.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182460 2024-03-25T23:59:03.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182460 2024-03-25T23:59:03.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673042 2-64424509513 3-313532612614 4-107374182460' 2024-03-25T23:59:03.747 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:03.747 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T23:59:03.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018934 2024-03-25T23:59:03.924 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018934 2024-03-25T23:59:03.924 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673042 2-64424509513 3-313532612614 4-107374182460 5-128849018934' 2024-03-25T23:59:03.924 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:03.924 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T23:59:04.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855401 2024-03-25T23:59:04.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855401 2024-03-25T23:59:04.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673042 2-64424509513 3-313532612614 4-107374182460 5-128849018934 6-150323855401' 2024-03-25T23:59:04.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:04.070 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-25T23:59:04.320 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691880 2024-03-25T23:59:04.321 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691880 2024-03-25T23:59:04.321 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673042 2-64424509513 3-313532612614 4-107374182460 5-128849018934 6-150323855401 7-171798691880' 2024-03-25T23:59:04.321 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:04.321 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-25T23:59:04.488 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528350 2024-03-25T23:59:04.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528350 2024-03-25T23:59:04.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673042 2-64424509513 3-313532612614 4-107374182460 5-128849018934 6-150323855401 7-171798691880 8-193273528350' 2024-03-25T23:59:04.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:04.489 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-25T23:59:04.642 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=335007449090 2024-03-25T23:59:04.642 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 335007449090 2024-03-25T23:59:04.642 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673042 2-64424509513 3-313532612614 4-107374182460 5-128849018934 6-150323855401 7-171798691880 8-193273528350 9-335007449090' 2024-03-25T23:59:04.642 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:04.642 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836573 2024-03-25T23:59:04.643 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:04.644 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:59:04.644 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836573 2024-03-25T23:59:04.644 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:04.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836573 2024-03-25T23:59:04.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836573' 2024-03-25T23:59:04.646 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 21474836573 2024-03-25T23:59:04.646 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:59:04.953 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836573 -lt 21474836573 2024-03-25T23:59:04.953 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:04.953 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673042 2024-03-25T23:59:04.954 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:04.955 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:59:04.956 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673042 2024-03-25T23:59:04.956 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:04.957 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673042 2024-03-25T23:59:04.957 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673042' 2024-03-25T23:59:04.957 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 42949673042 2024-03-25T23:59:04.958 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:59:05.295 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673042 -lt 42949673042 2024-03-25T23:59:05.295 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:05.295 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509513 2024-03-25T23:59:05.296 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:05.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:59:05.298 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509513 2024-03-25T23:59:05.298 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:05.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509513 2024-03-25T23:59:05.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509513' 2024-03-25T23:59:05.299 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 64424509513 2024-03-25T23:59:05.300 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:59:05.622 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509513 -lt 64424509513 2024-03-25T23:59:05.622 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:05.623 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-313532612614 2024-03-25T23:59:05.623 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:05.625 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:59:05.625 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-313532612614 2024-03-25T23:59:05.626 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:05.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612614 2024-03-25T23:59:05.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 313532612614' 2024-03-25T23:59:05.627 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.3 seq 313532612614 2024-03-25T23:59:05.627 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:59:05.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612615 -lt 313532612614 2024-03-25T23:59:05.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:05.963 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182460 2024-03-25T23:59:05.963 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:05.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T23:59:05.965 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182460 2024-03-25T23:59:05.965 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:05.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182460 2024-03-25T23:59:05.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182460' 2024-03-25T23:59:05.967 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.4 seq 107374182460 2024-03-25T23:59:05.967 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T23:59:06.264 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182460 -lt 107374182460 2024-03-25T23:59:06.264 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:06.265 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018934 2024-03-25T23:59:06.265 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:06.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T23:59:06.268 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018934 2024-03-25T23:59:06.268 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:06.269 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018934 2024-03-25T23:59:06.269 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018934' 2024-03-25T23:59:06.269 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.5 seq 128849018934 2024-03-25T23:59:06.269 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T23:59:06.666 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018934 -lt 128849018934 2024-03-25T23:59:06.666 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:06.666 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855401 2024-03-25T23:59:06.666 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:06.668 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-25T23:59:06.669 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855401 2024-03-25T23:59:06.669 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:06.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855401 2024-03-25T23:59:06.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855401' 2024-03-25T23:59:06.670 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.6 seq 150323855401 2024-03-25T23:59:06.671 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-25T23:59:07.055 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855402 -lt 150323855401 2024-03-25T23:59:07.055 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:07.055 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691880 2024-03-25T23:59:07.056 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:07.057 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-25T23:59:07.058 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691880 2024-03-25T23:59:07.058 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:07.059 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691880 2024-03-25T23:59:07.059 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691880' 2024-03-25T23:59:07.060 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.7 seq 171798691880 2024-03-25T23:59:07.060 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-25T23:59:07.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691880 -lt 171798691880 2024-03-25T23:59:07.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:07.460 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528350 2024-03-25T23:59:07.461 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:07.462 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-25T23:59:07.463 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528350 2024-03-25T23:59:07.463 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:07.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528350 2024-03-25T23:59:07.465 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528350' 2024-03-25T23:59:07.465 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.8 seq 193273528350 2024-03-25T23:59:07.465 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-25T23:59:07.791 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528350 -lt 193273528350 2024-03-25T23:59:07.792 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:07.792 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-335007449090 2024-03-25T23:59:07.792 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:07.794 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-25T23:59:07.794 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-335007449090 2024-03-25T23:59:07.794 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:07.796 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=335007449090 2024-03-25T23:59:07.796 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 335007449090' 2024-03-25T23:59:07.796 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.9 seq 335007449090 2024-03-25T23:59:07.796 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-25T23:59:08.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 335007449091 -lt 335007449090 2024-03-25T23:59:08.170 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:59:08.170 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:59:08.171 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:59:08.594 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:59:08.594 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:59:08.595 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:59:08.595 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:59:08.595 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:59:08.595 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:59:08.596 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:59:08.596 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:59:08.903 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:59:08.903 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:59:08.904 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:59:08.904 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:59:09.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:59:09.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:59:09.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:59:09.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-25T23:59:09.324 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-25T23:59:09.327 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2024-03-25T23:59:09.327 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-25T23:59:09.327 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-03-25T23:59:09.327 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=5 2024-03-25T23:59:09.327 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-25T23:59:09.327 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:59:09.327 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:59:09.327 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-25T23:59:09.327 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:59:09.327 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-03-25T23:59:09.328 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:59:09.328 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:59:09.328 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-25T23:59:09.328 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:59:09.328 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-25T23:59:09.328 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:59:09.328 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:59:09.328 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:59:09.328 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:59:09.328 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:59:09.636 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:59:09.636 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-03-25T23:59:09.636 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:59:09.636 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:59:09.636 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-25T23:59:09.637 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:59:09.637 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-25T23:59:09.637 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-03-25T23:59:11.927 INFO:tasks.workunit.client.0.smithi129.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-03-25T23:59:12.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-25T23:59:12.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:59:12.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:59:12.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-25T23:59:12.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:59:12.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-25T23:59:12.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:59:12.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:59:12.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:59:12.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:59:12.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-25T23:59:12.465 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-25T23:59:12.465 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:59:12.465 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:59:12.465 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:59:12.466 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:59:12.466 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:59:12.466 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:59:12.467 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:59:12.467 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:59:12.467 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:59:12.467 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:59:12.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:59:12.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:59:12.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:59:12.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:59:12.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:59:12.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:59:12.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:59:12.468 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:59:12.468 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:59:12.468 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:59:12.468 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-25T23:59:12.468 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-25T23:59:12.468 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.5 2024-03-25T23:59:12.469 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:59:12.469 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-25T23:59:12.469 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-25T23:59:12.470 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:59:12.470 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:59:12.470 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:59:12.503 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:59:12.504+0000 7f01d8a757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:59:12.531 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:59:12.532+0000 7f01d8a757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:59:12.532 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:59:12.532+0000 7f01d8a757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:59:12.788 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-25T23:59:12.788 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:59:12.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-25T23:59:12.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:59:12.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:59:12.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:59:12.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:59:12.789 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-25T23:59:12.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:59:12.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:59:13.076 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:59:13.076+0000 7f01d8a757c0 -1 Falling back to public interface 2024-03-25T23:59:13.133 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:59:14.134 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:59:14.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:59:14.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:59:14.135 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:59:14.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:59:14.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:59:14.441 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:59:15.443 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:59:15.443 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:59:15.443 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:59:15.443 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:59:15.444 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:59:15.444 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:59:15.702 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:59:15.704+0000 7f01d8a757c0 -1 osd.5 80 log_to_monitors true 2024-03-25T23:59:15.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:59:16.768 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:59:16.769 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:59:16.769 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:59:16.769 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:59:16.769 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:59:16.769 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:59:17.332 INFO:tasks.workunit.client.0.smithi129.stderr:osd.5 up in weight 1 up_from 84 up_thru 84 down_at 81 last_clean_interval [30,80) [v2:127.0.0.1:6842/972882195,v1:127.0.0.1:6843/972882195] [v2:127.0.0.1:6844/972882195,v1:127.0.0.1:6845/972882195] exists,up 67160811-ca71-46db-8586-da42b099b746 2024-03-25T23:59:17.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:59:17.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:59:17.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:59:17.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:59:17.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:59:17.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:59:17.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:59:17.332 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:59:17.333 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:59:17.333 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:59:17.333 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:59:17.333 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:59:17.333 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:59:17.441 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:59:17.442 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:59:17.442 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:59:17.442 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:59:17.442 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:59:17.442 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:59:17.864 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:59:17.864 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:59:17.864 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:59:17.865 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:59:17.865 INFO:tasks.workunit.client.0.smithi129.stderr:4 2024-03-25T23:59:17.865 INFO:tasks.workunit.client.0.smithi129.stderr:5 2024-03-25T23:59:17.865 INFO:tasks.workunit.client.0.smithi129.stderr:6 2024-03-25T23:59:17.865 INFO:tasks.workunit.client.0.smithi129.stderr:7 2024-03-25T23:59:17.865 INFO:tasks.workunit.client.0.smithi129.stderr:8 2024-03-25T23:59:17.865 INFO:tasks.workunit.client.0.smithi129.stderr:9' 2024-03-25T23:59:17.865 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:59:17.865 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:17.865 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:59:18.318 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836577 2024-03-25T23:59:18.318 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836577 2024-03-25T23:59:18.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577' 2024-03-25T23:59:18.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:18.319 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:59:18.538 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673046 2024-03-25T23:59:18.539 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673046 2024-03-25T23:59:18.539 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673046' 2024-03-25T23:59:18.539 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:18.539 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:59:18.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509517 2024-03-25T23:59:18.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509517 2024-03-25T23:59:18.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673046 2-64424509517' 2024-03-25T23:59:18.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:18.688 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:59:18.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612618 2024-03-25T23:59:18.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612618 2024-03-25T23:59:18.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673046 2-64424509517 3-313532612618' 2024-03-25T23:59:18.888 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:18.888 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T23:59:19.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182464 2024-03-25T23:59:19.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182464 2024-03-25T23:59:19.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673046 2-64424509517 3-313532612618 4-107374182464' 2024-03-25T23:59:19.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:19.032 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T23:59:19.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=360777252866 2024-03-25T23:59:19.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 360777252866 2024-03-25T23:59:19.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673046 2-64424509517 3-313532612618 4-107374182464 5-360777252866' 2024-03-25T23:59:19.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:19.178 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T23:59:19.315 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855405 2024-03-25T23:59:19.315 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855405 2024-03-25T23:59:19.315 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673046 2-64424509517 3-313532612618 4-107374182464 5-360777252866 6-150323855405' 2024-03-25T23:59:19.316 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:19.316 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-25T23:59:19.450 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691884 2024-03-25T23:59:19.450 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691884 2024-03-25T23:59:19.450 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673046 2-64424509517 3-313532612618 4-107374182464 5-360777252866 6-150323855405 7-171798691884' 2024-03-25T23:59:19.450 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:19.450 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-25T23:59:19.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528354 2024-03-25T23:59:19.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528354 2024-03-25T23:59:19.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673046 2-64424509517 3-313532612618 4-107374182464 5-360777252866 6-150323855405 7-171798691884 8-193273528354' 2024-03-25T23:59:19.589 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:19.589 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-25T23:59:19.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=335007449094 2024-03-25T23:59:19.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 335007449094 2024-03-25T23:59:19.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673046 2-64424509517 3-313532612618 4-107374182464 5-360777252866 6-150323855405 7-171798691884 8-193273528354 9-335007449094' 2024-03-25T23:59:19.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:19.728 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836577 2024-03-25T23:59:19.728 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:19.728 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:59:19.729 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836577 2024-03-25T23:59:19.729 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:19.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836577 2024-03-25T23:59:19.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836577' 2024-03-25T23:59:19.730 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 21474836577 2024-03-25T23:59:19.731 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:59:20.020 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836578 -lt 21474836577 2024-03-25T23:59:20.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:20.021 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673046 2024-03-25T23:59:20.022 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:20.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:59:20.024 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673046 2024-03-25T23:59:20.024 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:20.025 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673046 2024-03-25T23:59:20.025 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673046' 2024-03-25T23:59:20.025 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 42949673046 2024-03-25T23:59:20.026 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:59:20.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673046 -lt 42949673046 2024-03-25T23:59:20.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:20.351 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509517 2024-03-25T23:59:20.351 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:20.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:59:20.353 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509517 2024-03-25T23:59:20.353 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:20.355 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509517 2024-03-25T23:59:20.355 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509517' 2024-03-25T23:59:20.355 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 64424509517 2024-03-25T23:59:20.355 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:59:20.666 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509518 -lt 64424509517 2024-03-25T23:59:20.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:20.667 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-313532612618 2024-03-25T23:59:20.667 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:20.669 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:59:20.670 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-313532612618 2024-03-25T23:59:20.670 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:20.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612618 2024-03-25T23:59:20.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 313532612618' 2024-03-25T23:59:20.672 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.3 seq 313532612618 2024-03-25T23:59:20.672 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:59:20.983 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612618 -lt 313532612618 2024-03-25T23:59:20.984 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:20.984 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182464 2024-03-25T23:59:20.984 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:20.986 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T23:59:20.986 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182464 2024-03-25T23:59:20.987 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:20.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182464 2024-03-25T23:59:20.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182464' 2024-03-25T23:59:20.988 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.4 seq 107374182464 2024-03-25T23:59:20.988 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T23:59:21.317 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182464 -lt 107374182464 2024-03-25T23:59:21.317 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:21.317 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:21.317 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-360777252866 2024-03-25T23:59:21.318 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T23:59:21.319 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-360777252866 2024-03-25T23:59:21.319 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:21.321 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=360777252866 2024-03-25T23:59:21.321 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 360777252866' 2024-03-25T23:59:21.321 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.5 seq 360777252866 2024-03-25T23:59:21.322 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T23:59:21.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252866 -lt 360777252866 2024-03-25T23:59:21.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:21.665 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855405 2024-03-25T23:59:21.665 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:21.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-25T23:59:21.668 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:21.668 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855405 2024-03-25T23:59:21.669 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855405 2024-03-25T23:59:21.669 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855405' 2024-03-25T23:59:21.669 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.6 seq 150323855405 2024-03-25T23:59:21.670 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-25T23:59:22.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855406 -lt 150323855405 2024-03-25T23:59:22.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:22.018 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691884 2024-03-25T23:59:22.018 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:22.020 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-25T23:59:22.020 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691884 2024-03-25T23:59:22.021 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:22.022 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691884 2024-03-25T23:59:22.022 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691884' 2024-03-25T23:59:22.022 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.7 seq 171798691884 2024-03-25T23:59:22.022 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-25T23:59:22.333 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691884 -lt 171798691884 2024-03-25T23:59:22.333 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:22.334 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528354 2024-03-25T23:59:22.334 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:22.335 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-25T23:59:22.336 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:22.336 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528354 2024-03-25T23:59:22.338 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528354 2024-03-25T23:59:22.338 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528354' 2024-03-25T23:59:22.338 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.8 seq 193273528354 2024-03-25T23:59:22.338 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-25T23:59:22.684 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528354 -lt 193273528354 2024-03-25T23:59:22.684 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:22.685 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-335007449094 2024-03-25T23:59:22.685 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:22.687 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-25T23:59:22.688 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-335007449094 2024-03-25T23:59:22.688 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:22.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=335007449094 2024-03-25T23:59:22.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 335007449094' 2024-03-25T23:59:22.689 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.9 seq 335007449094 2024-03-25T23:59:22.689 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-25T23:59:23.028 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 335007449095 -lt 335007449094 2024-03-25T23:59:23.029 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:59:23.029 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:59:23.029 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:59:23.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:59:23.421 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:59:23.421 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:59:23.421 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:59:23.421 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:59:23.422 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:59:23.422 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:59:23.422 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:59:23.768 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:59:23.769 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:59:23.769 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:59:23.769 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:59:24.149 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:59:24.149 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:59:24.149 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:59:24.150 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-03-25T23:59:24.150 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-25T23:59:24.150 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-25T23:59:24.150 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-25T23:59:24.150 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-25T23:59:24.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-03-25T23:59:24.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-03-25T23:59:24.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-25T23:59:24.507 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-25T23:59:24.507 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:59:24.507 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:59:24.507 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:59:24.508 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:59:24.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-25T23:58:54.331151+0000 2024-03-25T23:59:24.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-25T23:59:25.152 INFO:tasks.workunit.client.0.smithi129.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-25T23:59:25.167 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-25T23:58:54.331151+0000 2024-03-25T23:59:25.167 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-25T23:59:25.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-25T23:58:54.331151+0000 2024-03-25T23:59:25.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-25T23:59:25.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-25T23:59:25.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:59:25.168 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:59:25.169 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:59:25.169 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:59:25.169 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:59:25.169 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:59:25.480 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:58:54.331151+0000 '>' 2024-03-25T23:58:54.331151+0000 2024-03-25T23:59:25.480 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-25T23:59:26.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-25T23:59:26.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-25T23:59:26.482 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-25T23:59:26.482 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:59:26.482 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:59:26.483 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:59:26.483 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:59:26.807 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:59:25.401442+0000 '>' 2024-03-25T23:58:54.331151+0000 2024-03-25T23:59:26.807 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-25T23:59:26.807 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-25T23:59:26.807 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:59:26.807 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:59:26.807 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-03-25T23:59:26.807 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:59:26.807 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-25T23:59:26.807 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:59:26.808 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:59:26.808 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-03-25T23:59:26.808 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:59:26.808 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-03-25T23:59:26.808 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:59:26.808 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:59:26.808 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:59:26.808 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:59:26.808 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:59:27.116 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:59:27.116 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-25T23:59:27.117 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:59:27.117 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:59:27.117 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-03-25T23:59:27.117 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:59:27.117 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-03-25T23:59:27.117 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-03-25T23:59:27.365 INFO:tasks.workunit.client.0.smithi129.stdout:_ 2024-03-25T23:59:27.365 INFO:tasks.workunit.client.0.smithi129.stdout:hinfo_key 2024-03-25T23:59:27.365 INFO:tasks.workunit.client.0.smithi129.stdout:snapset 2024-03-25T23:59:27.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-03-25T23:59:27.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:59:27.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:59:27.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-03-25T23:59:27.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:59:27.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-03-25T23:59:27.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:59:27.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:59:27.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:59:27.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:59:27.673 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-25T23:59:27.673 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-25T23:59:27.673 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:59:27.674 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:59:27.674 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:59:27.674 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:59:27.674 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:59:27.674 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:59:27.675 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:59:27.676 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:59:27.676 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:59:27.676 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:59:27.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:59:27.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:59:27.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:59:27.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:59:27.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:59:27.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:59:27.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:59:27.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:59:27.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:59:27.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:59:27.677 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-03-25T23:59:27.677 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-03-25T23:59:27.677 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.9 2024-03-25T23:59:27.677 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:59:27.677 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-03-25T23:59:27.677 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-03-25T23:59:27.678 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:59:27.678 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:59:27.678 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:59:27.708 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:59:27.708+0000 7f5e6de8f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:59:27.712 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:59:27.712+0000 7f5e6de8f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:59:27.713 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:59:27.712+0000 7f5e6de8f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:59:28.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-03-25T23:59:28.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:59:28.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-25T23:59:28.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:59:28.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:59:28.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:59:28.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:59:28.010 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-25T23:59:28.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:59:28.011 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:59:28.252 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:59:28.252+0000 7f5e6de8f7c0 -1 Falling back to public interface 2024-03-25T23:59:28.554 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:59:29.556 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:59:29.620 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:59:29.620 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:59:29.620 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:59:29.621 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:59:29.621 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:59:29.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:59:30.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:59:30.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:59:30.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:59:30.900 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:59:30.900 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:59:30.900 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:59:31.222 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:59:31.574 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-25T23:59:31.576+0000 7f5e6de8f7c0 -1 osd.9 86 log_to_monitors true 2024-03-25T23:59:32.224 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:59:32.224 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:59:32.224 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:59:32.224 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:59:32.224 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:59:32.224 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:59:32.616 INFO:tasks.workunit.client.0.smithi129.stderr:osd.9 up in weight 1 up_from 90 up_thru 0 down_at 87 last_clean_interval [78,86) [v2:127.0.0.1:6874/4233556636,v1:127.0.0.1:6875/4233556636] [v2:127.0.0.1:6876/4233556636,v1:127.0.0.1:6877/4233556636] exists,up 5fb70f5f-d7d5-45c0-b9fd-86c8b6646430 2024-03-25T23:59:32.617 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:59:32.617 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:59:32.617 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:59:32.617 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:59:32.617 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:59:32.617 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:59:32.617 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:59:32.617 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:59:32.618 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:59:32.618 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:59:32.618 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:59:32.618 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:59:32.618 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:59:32.728 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:59:32.728 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:59:32.728 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:59:32.728 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:59:32.728 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:59:32.728 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:59:33.410 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:59:33.411 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-25T23:59:33.411 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-25T23:59:33.411 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-25T23:59:33.411 INFO:tasks.workunit.client.0.smithi129.stderr:4 2024-03-25T23:59:33.411 INFO:tasks.workunit.client.0.smithi129.stderr:5 2024-03-25T23:59:33.411 INFO:tasks.workunit.client.0.smithi129.stderr:6 2024-03-25T23:59:33.411 INFO:tasks.workunit.client.0.smithi129.stderr:7 2024-03-25T23:59:33.411 INFO:tasks.workunit.client.0.smithi129.stderr:8 2024-03-25T23:59:33.411 INFO:tasks.workunit.client.0.smithi129.stderr:9' 2024-03-25T23:59:33.411 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:59:33.411 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:33.411 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:59:33.557 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836581 2024-03-25T23:59:33.558 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836581 2024-03-25T23:59:33.558 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581' 2024-03-25T23:59:33.558 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:33.558 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:59:33.904 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673050 2024-03-25T23:59:33.905 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673050 2024-03-25T23:59:33.905 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673050' 2024-03-25T23:59:33.905 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:33.905 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:59:34.287 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509521 2024-03-25T23:59:34.287 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509521 2024-03-25T23:59:34.287 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673050 2-64424509521' 2024-03-25T23:59:34.287 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:34.287 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:59:34.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612622 2024-03-25T23:59:34.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612622 2024-03-25T23:59:34.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673050 2-64424509521 3-313532612622' 2024-03-25T23:59:34.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:34.585 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T23:59:34.733 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182468 2024-03-25T23:59:34.733 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182468 2024-03-25T23:59:34.733 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673050 2-64424509521 3-313532612622 4-107374182468' 2024-03-25T23:59:34.733 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:34.733 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T23:59:34.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=360777252870 2024-03-25T23:59:34.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 360777252870 2024-03-25T23:59:34.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673050 2-64424509521 3-313532612622 4-107374182468 5-360777252870' 2024-03-25T23:59:34.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:34.872 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T23:59:35.020 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855409 2024-03-25T23:59:35.020 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855409 2024-03-25T23:59:35.020 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673050 2-64424509521 3-313532612622 4-107374182468 5-360777252870 6-150323855409' 2024-03-25T23:59:35.020 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:35.020 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-25T23:59:35.171 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691888 2024-03-25T23:59:35.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691888 2024-03-25T23:59:35.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673050 2-64424509521 3-313532612622 4-107374182468 5-360777252870 6-150323855409 7-171798691888' 2024-03-25T23:59:35.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:35.172 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-25T23:59:35.312 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528358 2024-03-25T23:59:35.313 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528358 2024-03-25T23:59:35.313 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673050 2-64424509521 3-313532612622 4-107374182468 5-360777252870 6-150323855409 7-171798691888 8-193273528358' 2024-03-25T23:59:35.313 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:35.313 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-25T23:59:35.455 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=386547056642 2024-03-25T23:59:35.456 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 386547056642 2024-03-25T23:59:35.456 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673050 2-64424509521 3-313532612622 4-107374182468 5-360777252870 6-150323855409 7-171798691888 8-193273528358 9-386547056642' 2024-03-25T23:59:35.456 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:35.456 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836581 2024-03-25T23:59:35.456 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:35.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:59:35.458 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836581 2024-03-25T23:59:35.458 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:35.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836581 2024-03-25T23:59:35.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836581' 2024-03-25T23:59:35.459 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 21474836581 2024-03-25T23:59:35.459 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:59:35.754 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836581 -lt 21474836581 2024-03-25T23:59:35.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:35.756 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673050 2024-03-25T23:59:35.756 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:35.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:59:35.757 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673050 2024-03-25T23:59:35.758 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:35.759 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673050 2024-03-25T23:59:35.759 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673050' 2024-03-25T23:59:35.759 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 42949673050 2024-03-25T23:59:35.759 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:59:36.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673050 -lt 42949673050 2024-03-25T23:59:36.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:36.124 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:36.125 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509521 2024-03-25T23:59:36.126 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:59:36.127 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509521 2024-03-25T23:59:36.127 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:36.128 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509521 2024-03-25T23:59:36.128 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509521' 2024-03-25T23:59:36.128 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 64424509521 2024-03-25T23:59:36.128 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:59:36.455 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509522 -lt 64424509521 2024-03-25T23:59:36.455 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:36.456 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-313532612622 2024-03-25T23:59:36.456 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:36.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:59:36.458 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-313532612622 2024-03-25T23:59:36.458 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:36.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612622 2024-03-25T23:59:36.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 313532612622' 2024-03-25T23:59:36.459 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.3 seq 313532612622 2024-03-25T23:59:36.460 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:59:36.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612623 -lt 313532612622 2024-03-25T23:59:36.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:36.783 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182468 2024-03-25T23:59:36.783 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:36.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T23:59:36.785 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182468 2024-03-25T23:59:36.785 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:36.787 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182468 2024-03-25T23:59:36.787 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182468' 2024-03-25T23:59:36.787 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.4 seq 107374182468 2024-03-25T23:59:36.787 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T23:59:37.111 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182468 -lt 107374182468 2024-03-25T23:59:37.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:37.112 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-360777252870 2024-03-25T23:59:37.112 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:37.115 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T23:59:37.115 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-360777252870 2024-03-25T23:59:37.116 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:37.117 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=360777252870 2024-03-25T23:59:37.117 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 360777252870' 2024-03-25T23:59:37.117 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.5 seq 360777252870 2024-03-25T23:59:37.118 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T23:59:37.451 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252870 -lt 360777252870 2024-03-25T23:59:37.451 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:37.452 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855409 2024-03-25T23:59:37.452 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:37.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-25T23:59:37.454 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855409 2024-03-25T23:59:37.454 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:37.455 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855409 2024-03-25T23:59:37.456 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855409' 2024-03-25T23:59:37.456 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.6 seq 150323855409 2024-03-25T23:59:37.456 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-25T23:59:37.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855410 -lt 150323855409 2024-03-25T23:59:37.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:37.780 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691888 2024-03-25T23:59:37.780 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:37.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-25T23:59:37.783 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691888 2024-03-25T23:59:37.783 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:37.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691888 2024-03-25T23:59:37.785 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691888' 2024-03-25T23:59:37.785 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.7 seq 171798691888 2024-03-25T23:59:37.785 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-25T23:59:38.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691888 -lt 171798691888 2024-03-25T23:59:38.163 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:38.164 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528358 2024-03-25T23:59:38.164 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:38.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-25T23:59:38.166 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528358 2024-03-25T23:59:38.166 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:38.167 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528358 2024-03-25T23:59:38.167 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528358' 2024-03-25T23:59:38.167 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.8 seq 193273528358 2024-03-25T23:59:38.168 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-25T23:59:38.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528358 -lt 193273528358 2024-03-25T23:59:38.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:38.505 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-386547056642 2024-03-25T23:59:38.505 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:38.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-25T23:59:38.508 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-386547056642 2024-03-25T23:59:38.508 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:38.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=386547056642 2024-03-25T23:59:38.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 386547056642' 2024-03-25T23:59:38.511 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.9 seq 386547056642 2024-03-25T23:59:38.511 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-25T23:59:38.807 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 386547056643 -lt 386547056642 2024-03-25T23:59:38.808 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:59:38.808 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:59:38.808 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:59:39.210 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:59:39.210 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:59:39.210 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:59:39.211 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:59:39.211 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:59:39.211 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:59:39.211 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:59:39.211 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:59:39.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T23:59:39.541 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:59:39.542 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:59:39.542 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:59:39.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-25T23:59:39.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-25T23:59:39.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-25T23:59:39.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-25T23:59:39.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T23:59:39.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-25T23:59:40.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T23:59:40.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:59:40.016 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:59:40.016 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:59:40.016 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:59:40.016 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:59:40.017 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:59:40.017 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:59:40.312 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:59:40.312 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:59:40.313 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:59:40.313 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:59:40.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:59:40.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:59:40.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:59:40.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-25T23:59:40.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-25T23:59:40.709 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2024-03-25T23:59:40.709 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-03-25T23:59:40.709 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-03-25T23:59:40.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=5 2024-03-25T23:59:40.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=9 2024-03-25T23:59:40.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-03-25T23:59:40.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-25T23:59:40.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:59:40.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:59:40.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 258079"' 2024-03-25T23:59:40.710 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 258079' 2024-03-25T23:59:40.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-03-25T23:59:40.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-25T23:59:40.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-25T23:59:40.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-25T23:59:40.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/258081: /' 2024-03-25T23:59:40.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 258082"' 2024-03-25T23:59:40.711 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 258082' 2024-03-25T23:59:40.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-03-25T23:59:40.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 258079 258082' 2024-03-25T23:59:40.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-25T23:59:40.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-25T23:59:40.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 258079 2024-03-25T23:59:40.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-03-25T23:59:40.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/258085: /' 2024-03-25T23:59:44.199 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:59:44.199 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:59:44.199 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-03-25T23:59:44.199 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:59:44.199 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2024-03-25T23:59:44.199 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:59:44.200 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:59:44.200 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-03-25T23:59:44.200 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:59:44.200 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-03-25T23:59:44.200 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:59:44.200 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:59:44.200 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:59:44.200 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:59:44.200 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:59:44.200 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:59:44.200 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2024-03-25T23:59:44.200 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:59:44.200 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:59:44.200 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-03-25T23:59:44.201 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:59:44.201 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-03-25T23:59:44.201 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2024-03-25T23:59:44.201 INFO:tasks.workunit.client.0.smithi129.stderr:258085: remove 2#2:eb822e21:::SOMETHING:head# 2024-03-25T23:59:44.201 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-03-25T23:59:44.201 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:59:44.201 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:59:44.201 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-03-25T23:59:44.201 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:59:44.201 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-03-25T23:59:44.201 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:59:44.201 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:59:44.204 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:59:44.204 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:59:44.204 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-25T23:59:44.204 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-25T23:59:44.204 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:59:44.204 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:59:44.204 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:59:44.204 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:59:44.204 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:59:44.205 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:59:44.205 INFO:tasks.workunit.client.0.smithi129.stderr:258085: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:59:44.205 INFO:tasks.workunit.client.0.smithi129.stderr:258085: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:59:44.205 INFO:tasks.workunit.client.0.smithi129.stderr:258085: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:59:44.205 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:59:44.205 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:59:44.205 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:59:44.205 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:59:44.205 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:59:44.205 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:59:44.205 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:59:44.205 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:59:44.205 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:59:44.205 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:59:44.206 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:59:44.206 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-03-25T23:59:44.206 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-03-25T23:59:44.206 INFO:tasks.workunit.client.0.smithi129.stderr:258085: start osd.9 2024-03-25T23:59:44.206 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:59:44.255 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-25T23:59:44.255 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T23:59:44.255 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-25T23:59:44.255 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T23:59:44.255 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-03-25T23:59:44.255 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-25T23:59:44.255 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T23:59:44.255 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-25T23:59:44.255 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T23:59:44.255 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-25T23:59:44.256 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T23:59:44.256 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T23:59:44.256 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T23:59:44.256 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T23:59:44.256 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T23:59:44.256 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T23:59:44.256 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-03-25T23:59:44.256 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-25T23:59:44.256 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T23:59:44.256 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-25T23:59:44.256 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T23:59:44.256 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-25T23:59:44.256 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-03-25T23:59:44.257 INFO:tasks.workunit.client.0.smithi129.stderr:258081: remove 1#2:eb822e21:::SOMETHING:head# 2024-03-25T23:59:44.257 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-25T23:59:44.257 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-25T23:59:44.257 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T23:59:44.257 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-25T23:59:44.257 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T23:59:44.257 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-25T23:59:44.257 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-25T23:59:44.257 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T23:59:44.258 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T23:59:44.258 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T23:59:44.258 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-25T23:59:44.258 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-25T23:59:44.258 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T23:59:44.258 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T23:59:44.258 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-25T23:59:44.258 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T23:59:44.258 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T23:59:44.259 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T23:59:44.259 INFO:tasks.workunit.client.0.smithi129.stderr:258081: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T23:59:44.259 INFO:tasks.workunit.client.0.smithi129.stderr:258081: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T23:59:44.259 INFO:tasks.workunit.client.0.smithi129.stderr:258081: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-25T23:59:44.259 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:59:44.259 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-25T23:59:44.259 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T23:59:44.259 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-25T23:59:44.259 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-25T23:59:44.259 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T23:59:44.259 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T23:59:44.259 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T23:59:44.259 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T23:59:44.259 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T23:59:44.260 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T23:59:44.260 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-25T23:59:44.260 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-25T23:59:44.260 INFO:tasks.workunit.client.0.smithi129.stderr:258081: start osd.5 2024-03-25T23:59:44.260 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T23:59:49.107 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-03-25T23:59:49.107 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-03-25T23:59:49.107 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:59:49.107 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:59:49.107 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:59:49.107 INFO:tasks.workunit.client.0.smithi129.stderr:258085: 2024-03-25T23:59:44.235+0000 7fb3185a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:59:49.107 INFO:tasks.workunit.client.0.smithi129.stderr:258085: 2024-03-25T23:59:44.239+0000 7fb3185a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:59:49.107 INFO:tasks.workunit.client.0.smithi129.stderr:258085: 2024-03-25T23:59:44.239+0000 7fb3185a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:59:49.108 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-03-25T23:59:49.108 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:59:49.108 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-25T23:59:49.108 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:59:49.108 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:59:49.108 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:59:49.108 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:59:49.108 INFO:tasks.workunit.client.0.smithi129.stderr:258085: 0 2024-03-25T23:59:49.108 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:59:49.108 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:59:49.108 INFO:tasks.workunit.client.0.smithi129.stderr:258085: 2024-03-25T23:59:44.779+0000 7fb3185a47c0 -1 Falling back to public interface 2024-03-25T23:59:49.108 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:59:49.108 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:59:49.109 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:59:49.109 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:59:49.109 INFO:tasks.workunit.client.0.smithi129.stderr:258085: 1 2024-03-25T23:59:49.109 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:59:49.109 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:59:49.109 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:59:49.109 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:59:49.109 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:59:49.109 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:59:49.109 INFO:tasks.workunit.client.0.smithi129.stderr:258085: 2 2024-03-25T23:59:49.109 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:59:49.109 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:59:49.109 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:59:49.109 INFO:tasks.workunit.client.0.smithi129.stderr:258085: 2024-03-25T23:59:47.779+0000 7fb3185a47c0 -1 osd.9 92 log_to_monitors true 2024-03-25T23:59:49.110 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:59:49.110 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:59:49.110 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:59:49.110 INFO:tasks.workunit.client.0.smithi129.stderr:258085: 3 2024-03-25T23:59:49.110 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:59:49.110 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T23:59:49.110 INFO:tasks.workunit.client.0.smithi129.stderr:258085: osd.9 up in weight 1 up_from 96 up_thru 0 down_at 93 last_c258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-25T23:59:49.110 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-25T23:59:49.110 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T23:59:49.110 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T23:59:49.110 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T23:59:49.110 INFO:tasks.workunit.client.0.smithi129.stderr:258081: 2024-03-25T23:59:44.287+0000 7f580bb0a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:59:49.110 INFO:tasks.workunit.client.0.smithi129.stderr:258081: 2024-03-25T23:59:44.287+0000 7f580bb0a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:59:49.110 INFO:tasks.workunit.client.0.smithi129.stderr:258081: 2024-03-25T23:59:44.291+0000 7f580bb0a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T23:59:49.111 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-25T23:59:49.111 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T23:59:49.111 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-25T23:59:49.111 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T23:59:49.111 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T23:59:49.111 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:59:49.111 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T23:59:49.111 INFO:tasks.workunit.client.0.smithi129.stderr:258081: 0 2024-03-25T23:59:49.111 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:59:49.111 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:59:49.111 INFO:tasks.workunit.client.0.smithi129.stderr:258081: 2024-03-25T23:59:44.835+0000 7f580bb0a7c0 -1 Falling back to public interface 2024-03-25T23:59:49.111 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:59:49.111 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:59:49.111 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:59:49.112 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T23:59:49.112 INFO:tasks.workunit.client.0.smithi129.stderr:258081: 1 2024-03-25T23:59:49.112 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:59:49.112 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:59:49.112 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:59:49.112 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:59:49.112 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:59:49.112 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T23:59:49.112 INFO:tasks.workunit.client.0.smithi129.stderr:258081: 2 2024-03-25T23:59:49.112 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:59:49.112 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:59:49.112 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T23:59:49.112 INFO:tasks.workunit.client.0.smithi129.stderr:258081: 2024-03-25T23:59:48.183+0000 7f580bb0a7c0 -1 osd.5 92 log_to_monitors true 2024-03-25T23:59:49.112 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:59:49.113 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:59:49.113 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T23:59:49.113 INFO:tasks.workunit.client.0.smithi129.stderr:258081: 3 2024-03-25T23:59:49.113 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:59:49.113 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:59:50.600 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperlean_interval [90,92) [v2:127.0.0.1:6842/1614089254,v1:127.0.0.1:6843/1614089254] [v2:127.0.0.1:6844/1614089254,v1:127.0.0.1:6845/1614089254] exists,up 5fb70f5f-d7d5-45c0-b9fd-86c8b6646430 2024-03-25T23:59:50.600 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:59:50.600 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:59:50.600 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:59:50.600 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:59:50.600 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:59:50.601 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:59:50.601 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:59:50.601 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:59:50.601 INFO:tasks.workunit.client.0.smithi129.stderr:258085: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:59:50.601 INFO:tasks.workunit.client.0.smithi129.stderr:258085: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:59:50.601 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:59:50.601 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:59:50.601 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:59:50.601 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:59:50.601 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:59:50.601 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:59:50.601 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:59:50.601 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:59:50.601 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:59:50.602 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:59:50.602 INFO:tasks.workunit.client.0.smithi129.stderr:258085: 1 2024-03-25T23:59:50.602 INFO:tasks.workunit.client.0.smithi129.stderr:258085: 2 2024-03-25T23:59:50.602 INFO:tasks.workunit.client.0.smithi129.stderr:258085: 3 2024-03-25T23:59:50.602 INFO:tasks.workunit.client.0.smithi129.stderr:258085: 4 2024-03-25T23:59:50.602 INFO:tasks.workunit.client.0.smithi129.stderr:258085: 5 2024-03-25T23:59:50.602 INFO:tasks.workunit.client.0.smithi129.stderr:258085: 6 2024-03-25T23:59:50.602 INFO:tasks.workunit.client.0.smithi129.stderr:258085: 7 2024-03-25T23:59:50.602 INFO:tasks.workunit.client.0.smithi129.stderr:258085: 8 2024-03-25T23:59:50.602 INFO:tasks.workunit.client.0.smithi129.stderr:258085: 9' 2024-03-25T23:59:50.602 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:59:50.602 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:50.602 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:59:50.603 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836586 2024-03-25T23:59:50.603 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836586 2024-03-25T23:59:50.603 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586' 2024-03-25T23:59:50.603 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:50.603 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:59:50.603 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673054 2024-03-25T23:59:50.603 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673054 2024-03-25T23:59:50.603 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673054' 2024-03-25T23:59:50.603 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:51.851 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qas.sh:985: wait_for_osd: sleep 1 2024-03-25T23:59:51.851 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T23:59:51.851 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T23:59:51.851 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T23:59:51.851 INFO:tasks.workunit.client.0.smithi129.stderr:258081: 4 2024-03-25T23:59:51.851 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T23:59:51.851 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T23:59:51.852 INFO:tasks.workunit.client.0.smithi129.stderr:258081: osd.5 up in weight 1 up_from 98 up_thru 98 down_at 93 last_clean_interval [84,92) [v2:127.0.0.1:6874/3225168545,v1:127.0.0.1:6875/3225168545] [v2:127.0.0.1:6876/3225168545,v1:127.0.0.1:6877/3225168545] exists,up 67160811-ca71-46db-8586-da42b099b746 2024-03-25T23:59:51.852 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T23:59:51.852 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T23:59:51.852 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T23:59:51.852 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T23:59:51.852 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T23:59:51.852 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T23:59:51.852 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T23:59:51.852 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T23:59:51.852 INFO:tasks.workunit.client.0.smithi129.stderr:258081: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T23:59:51.852 INFO:tasks.workunit.client.0.smithi129.stderr:258081: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T23:59:51.852 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T23:59:51.852 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T23:59:51.852 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T23:59:51.853 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T23:59:51.853 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T23:59:51.853 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T23:59:51.853 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T23:59:51.853 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T23:59:51.853 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T23:59:51.853 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T23:59:51.853 INFO:tasks.workunit.client.0.smithi129.stderr:258081: 1 2024-03-25T23:59:51.853 INFO:tasks.workunit.client.0.smithi129.stderr:258081: 2 2024-03-25T23:59:51.853 INFO:tasks.workunit.client.0.smithi129.stderr:258081: 3 2024-03-25T23:59:51.853 INFO:tasks.workunit.client.0.smithi129.stderr:258081: 4 2024-03-25T23:59:51.853 INFO:tasks.workunit.client.0.smithi129.stderr:258081: 5 2024-03-25T23:59:51.853 INFO:tasks.workunit.client.0.smithi129.stderr:258081: 6 2024-03-25T23:59:51.853 INFO:tasks.workunit.client.0.smithi129.stderr:258081: 7 2024-03-25T23:59:51.853 INFO:tasks.workunit.client.0.smithi129.stderr:258081: 8 2024-03-25T23:59:51.854 INFO:tasks.workunit.client.0.smithi129.stderr:258081: 9' 2024-03-25T23:59:51.854 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T23:59:51.854 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:51.854 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T23:59:51.854 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836587 2024-03-25T23:59:51.854 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836587 2024-03-25T23:59:51.854 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587' 2024-03-25T23:59:51.854 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:52.539 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:59:52.539 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509526 2024-03-25T23:59:52.539 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509526 2024-03-25T23:59:52.539 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673054 2-64424509526' 2024-03-25T23:59:52.539 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:52.539 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:59:52.539 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612627 2024-03-25T23:59:52.539 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612627 2024-03-25T23:59:52.539 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673054 2-64424509526 3-313532612627' 2024-03-25T23:59:52.540 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:52.540 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T23:59:52.540 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182472 2024-03-25T23:59:52.540 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182472 2024-03-25T23:59:52.540 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673054 2-64424509526 3-313532612627 4-107374182472' 2024-03-25T23:59:52.540 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:52.540 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T23:59:52.540 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=420906795010 2024-03-25T23:59:52.540 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 420906795010 2024-03-25T23:59:52.540 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673054 2-64424509526 3-313532612627 4-107374182472 5-420906795010' 2024-03-25T23:59:52.540 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:52.540 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T23:59:52.540 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855414 2024-03-25T23:59:52.540 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855414 2024-03-25T23:59:52.540 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673054 2-64424509526 3-313532612627 4-107374182472 5-420906795010 6-150323855414' 2024-03-25T23:59:52.541 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:52.541 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-25T23:59:52.541 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691892 2024-03-25T23:59:52.541 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691892 2024-03-25T23:59:52.541 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673054 2-64424509526 3-313532612627 4-107374182472 5-420906795010 6-150323855414 7-171798691892' 2024-03-25T23:59:52.541 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:52.541 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-25T23:59:53.092 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone//home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T23:59:53.092 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673056 2024-03-25T23:59:53.092 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673056 2024-03-25T23:59:53.092 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673056' 2024-03-25T23:59:53.092 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:53.092 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T23:59:53.093 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509527 2024-03-25T23:59:53.093 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509527 2024-03-25T23:59:53.093 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673056 2-64424509527' 2024-03-25T23:59:53.093 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:53.093 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T23:59:53.093 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612628 2024-03-25T23:59:53.093 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612628 2024-03-25T23:59:53.093 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673056 2-64424509527 3-313532612628' 2024-03-25T23:59:53.093 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:53.093 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T23:59:53.093 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182473 2024-03-25T23:59:53.093 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182473 2024-03-25T23:59:53.093 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673056 2-64424509527 3-313532612628 4-107374182473' 2024-03-25T23:59:53.094 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:53.094 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T23:59:53.094 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=420906795011 2024-03-25T23:59:53.094 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 420906795011 2024-03-25T23:59:53.094 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673056 2-64424509527 3-313532612628 4-107374182473 5-420906795011' 2024-03-25T23:59:53.094 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:53.094 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T23:59:53.094 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855415 2024-03-25T23:59:53.094 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855415 2024-03-25T23:59:53.094 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673056 2-64424509527 3-313532612628 4-107374182473 5-420906795011 6-150323855415' 2024-03-25T23:59:53.094 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:53.094 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-25T23:59:53.351 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528362 2024-03-25T23:59:53.351 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528362 2024-03-25T23:59:53.352 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673054 2-64424509526 3-313532612627 4-107374182472 5-420906795010 6-150323855414 7-171798691892 8-193273528362' 2024-03-25T23:59:53.352 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:53.352 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-25T23:59:53.352 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=412316860418 2024-03-25T23:59:53.352 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 412316860418 2024-03-25T23:59:53.352 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673054 2-64424509526 3-313532612627 4-107374182472 5-420906795010 6-150323855414 7-171798691892 8-193273528362 9-412316860418' 2024-03-25T23:59:53.352 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:53.352 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836586 2024-03-25T23:59:53.352 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:53.352 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:59:53.352 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836586 2024-03-25T23:59:53.352 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:53.352 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836586 2024-03-25T23:59:53.353 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836586' 2024-03-25T23:59:53.353 INFO:tasks.workunit.client.0.smithi129.stderr:258085: waiting osd.0 seq 21474836586 2024-03-25T23:59:53.353 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:59:53.353 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836586 -lt 21474836586 2024-03-25T23:59:53.353 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:53.353 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673054 2024-03-25T23:59:53.354 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:53.354 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:59:53.354 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673054 2024-03-25T23:59:53.354 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:53.354 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673054 2024-03-25T23:59:53.354 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673054' 2024-03-25T23:59:53.354 INFO:tasks.workunit.client.0.smithi129.stderr:258085: waiting osd.1 seq 42949673054 2024-03-25T23:59:53.354 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:59:53.354 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673055 -lt 42949673054 2024-03-25T23:59:53.354 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:53.355 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509526 2024-03-25T23:59:53.355 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:53.355 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:59:54.362 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509526 2024-03-25T23:59:54.362 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:54.362 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509526 2024-03-25T23:59:54.362 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509526' 2024-03-25T23:59:54.362 INFO:tasks.workunit.client.0.smithi129.stderr:258085: waiting osd.2 seq 64424509526 2024-03-25T23:59:54.362 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:59:54.362 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509526 -lt 64424509526 2024-03-25T23:59:54.363 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:54.363 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-313532612627 2024-03-25T23:59:54.363 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:54.363 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:59:54.363 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-313532612627 2024-03-25T23:59:54.363 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:54.363 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612627 2024-03-25T23:59:54.363 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 313532612627' 2024-03-25T23:59:54.363 INFO:tasks.workunit.client.0.smithi129.stderr:258085: waiting osd.3 seq 313532612627 2024-03-25T23:59:54.364 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:59:54.364 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612628 -lt 313532612627 2024-03-25T23:59:54.364 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:54.364 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182472 2024-03-25T23:59:54.364 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:54.364 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T23:59:54.364 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182472 2024-03-25T23:59:54.364 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:54.364 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182472 2024-03-25T23:59:54.364 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182472' 2024-03-25T23:59:54.365 INFO:tasks.workunit.client.0.smithi129.stderr:258085: waiting osd.4 seq 107374182472 2024-03-25T23:59:54.365 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T23:59:54.365 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182474 -lt 107374182472 2024-03-25T23:59:54.365 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:54.365 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-420906795010 2024-03-25T23:59:54.365 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:54.365 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T23:59:54.365 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-420906795010 2024-03-25T23:59:54.365 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:54.365 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=420906795010 2024-03-25T23:59:55.034 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiti_pg_stats: seq=171798691893 2024-03-25T23:59:55.034 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691893 2024-03-25T23:59:55.034 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673056 2-64424509527 3-313532612628 4-107374182473 5-420906795011 6-150323855415 7-171798691893' 2024-03-25T23:59:55.034 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:55.034 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-25T23:59:55.035 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528363 2024-03-25T23:59:55.035 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528363 2024-03-25T23:59:55.035 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673056 2-64424509527 3-313532612628 4-107374182473 5-420906795011 6-150323855415 7-171798691893 8-193273528363' 2024-03-25T23:59:55.035 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T23:59:55.035 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-25T23:59:55.035 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=412316860420 2024-03-25T23:59:55.035 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 412316860420 2024-03-25T23:59:55.035 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673056 2-64424509527 3-313532612628 4-107374182473 5-420906795011 6-150323855415 7-171798691893 8-193273528363 9-412316860420' 2024-03-25T23:59:55.035 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:55.035 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836587 2024-03-25T23:59:55.035 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:55.035 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T23:59:55.035 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836587 2024-03-25T23:59:55.036 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:55.036 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836587 2024-03-25T23:59:55.036 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836587' 2024-03-25T23:59:55.036 INFO:tasks.workunit.client.0.smithi129.stderr:258081: waiting osd.0 seq 21474836587 2024-03-25T23:59:55.036 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:59:55.036 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836586 -lt 21474836587 2024-03-25T23:59:55.036 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T23:59:55.036 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T23:59:55.036 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T23:59:55.036 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836587 -lt 21474836587 2024-03-25T23:59:55.036 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:55.036 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:55.037 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673056 2024-03-25T23:59:55.037 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T23:59:55.037 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673056 2024-03-25T23:59:55.715 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //homeng osd.5 seq 420906795010' 2024-03-25T23:59:55.716 INFO:tasks.workunit.client.0.smithi129.stderr:258085: waiting osd.5 seq 420906795010 2024-03-25T23:59:55.716 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T23:59:55.716 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 420906795011 -lt 420906795010 2024-03-25T23:59:55.716 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:55.716 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:55.716 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855414 2024-03-25T23:59:55.716 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-25T23:59:55.716 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:55.716 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855414 2024-03-25T23:59:55.716 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855414 2024-03-25T23:59:55.716 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855414' 2024-03-25T23:59:55.716 INFO:tasks.workunit.client.0.smithi129.stderr:258085: waiting osd.6 seq 150323855414 2024-03-25T23:59:55.717 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-25T23:59:55.717 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855415 -lt 150323855414 2024-03-25T23:59:55.717 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:55.717 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691892 2024-03-25T23:59:55.717 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:55.717 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-25T23:59:55.717 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691892 2024-03-25T23:59:55.717 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:55.717 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691892 2024-03-25T23:59:55.717 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691892' 2024-03-25T23:59:55.717 INFO:tasks.workunit.client.0.smithi129.stderr:258085: waiting osd.7 seq 171798691892 2024-03-25T23:59:55.717 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-25T23:59:55.717 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691894 -lt 171798691892 2024-03-25T23:59:55.717 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:55.717 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528362 2024-03-25T23:59:55.718 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:55.718 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-25T23:59:55.718 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528362 2024-03-25T23:59:55.718 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:55.718 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528362 2024-03-25T23:59:55.718 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528362' 2024-03-25T23:59:55.718 INFO:tasks.workunit.client.0.smithi129.stderr:258085: waiting osd.8 seq 193273528362 2024-03-25T23:59:55.718 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-25T23:59:55.718 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528363 -lt 193273528362 2024-03-25T23:59:56.113 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:56.113 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673056 2024-03-25T23:59:56.113 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673056' 2024-03-25T23:59:56.113 INFO:tasks.workunit.client.0.smithi129.stderr:258081: waiting osd.1 seq 42949673056 2024-03-25T23:59:56.113 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T23:59:56.113 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673056 -lt 42949673056 2024-03-25T23:59:56.114 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:56.114 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509527 2024-03-25T23:59:56.114 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:56.114 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T23:59:56.114 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509527 2024-03-25T23:59:56.114 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:56.114 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509527 2024-03-25T23:59:56.114 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509527' 2024-03-25T23:59:56.114 INFO:tasks.workunit.client.0.smithi129.stderr:258081: waiting osd.2 seq 64424509527 2024-03-25T23:59:56.114 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T23:59:56.114 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509527 -lt 64424509527 2024-03-25T23:59:56.114 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:56.114 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-313532612628 2024-03-25T23:59:56.115 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:56.115 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T23:59:56.115 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-313532612628 2024-03-25T23:59:56.115 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:56.115 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612628 2024-03-25T23:59:56.115 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 313532612628' 2024-03-25T23:59:56.115 INFO:tasks.workunit.client.0.smithi129.stderr:258081: waiting osd.3 seq 313532612628 2024-03-25T23:59:56.115 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T23:59:56.115 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612629 -lt 313532612628 2024-03-25T23:59:56.115 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:56.115 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182473 2024-03-25T23:59:56.115 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:56.115 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T23:59:56.115 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182473 2024-03-25T23:59:56.115 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:56.116 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182473 2024-03-25T23:59:56.116 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182473' 2024-03-25T23:59:56.116 INFO:tasks.workunit.client.0.smithi129.stderr:258081: waiting osd.4 seq 107374182473 2024-03-25T23:59:57.221 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clon_stats: for s in $seqs 2024-03-25T23:59:57.221 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-412316860418 2024-03-25T23:59:57.221 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:57.221 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-25T23:59:57.221 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-412316860418 2024-03-25T23:59:57.221 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:57.221 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=412316860418 2024-03-25T23:59:57.221 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 412316860418' 2024-03-25T23:59:57.221 INFO:tasks.workunit.client.0.smithi129.stderr:258085: waiting osd.9 seq 412316860418 2024-03-25T23:59:57.221 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-25T23:59:57.221 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 412316860420 -lt 412316860418 2024-03-25T23:59:57.221 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:59:57.222 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:59:57.222 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:59:57.222 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:59:57.222 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:59:57.222 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:59:57.222 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:59:57.222 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:59:57.222 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:59:57.222 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:59:57.222 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:59:57.222 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:59:57.222 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:59:57.222 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:59:57.223 INFO:tasks.workunit.client.0.smithi129.stderr:258085: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:59:57.223 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:59:57.223 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:59:57.223 INFO:tasks.workunit.client.0.smithi129.stderr:258085: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:59:57.223 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:59:57.369 INFO:tasks.workunit.client.0.smithi129.stderr:e.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T23:59:57.369 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182474 -lt 107374182473 2024-03-25T23:59:57.369 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:57.369 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-420906795011 2024-03-25T23:59:57.369 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:57.369 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T23:59:57.369 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-420906795011 2024-03-25T23:59:57.369 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:57.369 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=420906795011 2024-03-25T23:59:57.369 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 420906795011' 2024-03-25T23:59:57.369 INFO:tasks.workunit.client.0.smithi129.stderr:258081: waiting osd.5 seq 420906795011 2024-03-25T23:59:57.370 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T23:59:57.370 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 420906795012 -lt 420906795011 2024-03-25T23:59:57.370 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:57.370 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855415 2024-03-25T23:59:57.370 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:57.370 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-25T23:59:57.370 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855415 2024-03-25T23:59:57.370 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:57.370 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855415 2024-03-25T23:59:57.370 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855415' 2024-03-25T23:59:57.370 INFO:tasks.workunit.client.0.smithi129.stderr:258081: waiting osd.6 seq 150323855415 2024-03-25T23:59:57.370 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-25T23:59:57.370 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855416 -lt 150323855415 2024-03-25T23:59:57.370 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:57.371 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691893 2024-03-25T23:59:57.371 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:57.371 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-25T23:59:57.371 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691893 2024-03-25T23:59:57.371 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:57.371 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691893 2024-03-25T23:59:57.371 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691893' 2024-03-25T23:59:57.371 INFO:tasks.workunit.client.0.smithi129.stderr:258081: waiting osd.7 seq 171798691893 2024-03-25T23:59:57.371 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-25T23:59:57.371 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691894 -lt 171798691893 2024-03-25T23:59:57.371 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:58.714 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528363 2024-03-25T23:59:58.714 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:58.714 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-25T23:59:58.714 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528363 2024-03-25T23:59:58.715 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:58.715 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528363 2024-03-25T23:59:58.715 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528363' 2024-03-25T23:59:58.715 INFO:tasks.workunit.client.0.smithi129.stderr:258081: waiting osd.8 seq 193273528363 2024-03-25T23:59:58.715 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-25T23:59:58.715 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528364 -lt 193273528363 2024-03-25T23:59:58.715 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T23:59:58.715 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-412316860420 2024-03-25T23:59:58.715 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T23:59:58.715 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-25T23:59:58.715 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-412316860420 2024-03-25T23:59:58.715 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T23:59:58.716 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=412316860420 2024-03-25T23:59:58.716 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 412316860420' 2024-03-25T23:59:58.716 INFO:tasks.workunit.client.0.smithi129.stderr:258081: waiting osd.9 seq 412316860420 2024-03-25T23:59:58.716 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-25T23:59:58.716 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 412316860420 -lt 412316860420 2024-03-25T23:59:58.716 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T23:59:58.716 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:59:58.716 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:59:58.716 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T23:59:58.716 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T23:59:58.716 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T23:59:58.716 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T23:59:58.716 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T23:59:58.716 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T23:59:58.716 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T23:59:58.717 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T23:59:58.717 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T23:59:58.717 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T23:59:59.072 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T23:59:59.072 INFO:tasks.workunit.client.0.smithi129.stderr:258081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T23:59:59.072 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T23:59:59.073 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T23:59:59.073 INFO:tasks.workunit.client.0.smithi129.stderr:258081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T23:59:59.073 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-25T23:59:59.073 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-25T23:59:59.073 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 258082 2024-03-25T23:59:59.073 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-25T23:59:59.073 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-25T23:59:59.073 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-25T23:59:59.073 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-03-25T23:59:59.073 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-25T23:59:59.074 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-03-25T23:59:59.074 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-25T23:59:59.074 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-25T23:59:59.074 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-25T23:59:59.074 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-25T23:59:59.441 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-03-25T23:59:59.442 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-03-25T23:59:59.442 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-25T23:59:59.442 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-25T23:59:59.442 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-25T23:59:59.442 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-25T23:59:59.443 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-25T23:59:59.443 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-25T23:59:59.739 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-25T23:59:25.401442+0000 2024-03-25T23:59:59.739 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-26T00:00:00.037 INFO:tasks.workunit.client.0.smithi129.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-26T00:00:00.054 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-25T23:59:25.401442+0000 2024-03-26T00:00:00.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-26T00:00:00.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-25T23:59:25.401442+0000 2024-03-26T00:00:00.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-26T00:00:00.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-26T00:00:00.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:00:00.070 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-26T00:00:00.071 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-26T00:00:00.071 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:00:00.071 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:00:00.071 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-26T00:00:00.361 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:59:25.401442+0000 '>' 2024-03-25T23:59:25.401442+0000 2024-03-26T00:00:00.362 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:00:01.363 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:00:01.363 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:00:01.364 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-26T00:00:01.364 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-26T00:00:01.364 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:00:01.365 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:00:01.365 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-26T00:00:01.694 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T23:59:25.401442+0000 '>' 2024-03-25T23:59:25.401442+0000 2024-03-26T00:00:01.694 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:00:02.696 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:00:02.696 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:00:02.697 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-26T00:00:02.697 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-26T00:00:02.697 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:00:02.698 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:00:02.698 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-26T00:00:03.018 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:00:00.585529+0000 '>' 2024-03-25T23:59:25.401442+0000 2024-03-26T00:00:03.018 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-26T00:00:03.019 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-03-26T00:00:03.019 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-26T00:00:03.019 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-26T00:00:03.019 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-26T00:00:03.019 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 261482"' 2024-03-26T00:00:03.019 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 261482' 2024-03-26T00:00:03.019 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-26T00:00:03.019 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-26T00:00:03.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-26T00:00:03.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 261483"' 2024-03-26T00:00:03.023 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 261483' 2024-03-26T00:00:03.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-03-26T00:00:03.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 261482 261483' 2024-03-26T00:00:03.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-26T00:00:03.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-26T00:00:03.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 261482 2024-03-26T00:00:03.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-26T00:00:03.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-26T00:00:03.024 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/261486: /' 2024-03-26T00:00:03.024 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/261487: /' 2024-03-26T00:00:04.031 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:00:04.031 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:00:04.031 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-03-26T00:00:04.031 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:00:04.031 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-26T00:00:04.031 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:00:04.031 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:00:04.031 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-03-26T00:00:04.031 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:00:04.031 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-03-26T00:00:04.032 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:00:04.032 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:00:04.032 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:00:04.032 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:00:04.032 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:00:04.032 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:00:04.032 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-26T00:00:04.032 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:00:04.032 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:00:04.032 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-03-26T00:00:04.032 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:00:04.032 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-03-26T00:00:04.032 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-03-26T00:00:04.032 INFO:tasks.workunit.client.0.smithi129.stderr:261486: _ 2024-03-26T00:00:04.033 INFO:tasks.workunit.client.0.smithi129.stderr:261486: hinfo_key 2024-03-26T00:00:04.033 INFO:tasks.workunit.client.0.smithi129.stderr:261486: snapset 2024-03-26T00:00:04.033 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-03-26T00:00:04.033 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:00:04.033 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:00:04.033 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-03-26T00:00:04.033 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:00:04.033 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-03-26T00:00:04.033 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:00:04.033 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:00:04.034 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helper261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:00:04.034 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:00:04.034 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-26T00:00:04.034 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:00:04.034 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-26T00:00:04.034 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:00:04.034 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:00:04.034 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-26T00:00:04.034 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:00:04.035 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-26T00:00:04.035 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:00:04.035 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:00:04.035 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:00:04.035 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:00:04.035 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:00:04.035 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:00:04.035 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-26T00:00:04.035 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:00:04.035 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:00:04.035 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-26T00:00:04.035 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:00:04.035 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-26T00:00:04.035 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-03-26T00:00:04.036 INFO:tasks.workunit.client.0.smithi129.stderr:261487: _ 2024-03-26T00:00:04.036 INFO:tasks.workunit.client.0.smithi129.stderr:261487: hinfo_key 2024-03-26T00:00:04.036 INFO:tasks.workunit.client.0.smithi129.stderr:261487: snapset 2024-03-26T00:00:04.036 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-26T00:00:04.036 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:00:04.036 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:00:04.036 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-26T00:00:04.036 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:00:04.036 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-26T00:00:04.036 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:00:04.036 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:00:04.037 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:00:04.037 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:00:04.037 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-26T00:00:04.037 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-26T00:00:04.037 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:00:04.037 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:00:04.037 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:00:04.037 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:00:04.037 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:00:04.037 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:00:04.037 INFO:tasks.workunit.client.0.smithi129.stderr:261486: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:00:04.037 INFO:tasks.workunit.client.0.smithi129.stderr:261486: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:00:04.037 INFO:tasks.workunit.client.0.smithi129.stderr:261486: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:00:04.038 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:00:04.038 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:00:04.038 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:00:04.038 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:00:04.038 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:00:04.038 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:00:04.038 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:00:04.038 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:00:04.038 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:00:04.038 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:00:04.038 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:00:04.038 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-03-26T00:00:04.038 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-03-26T00:00:04.038 INFO:tasks.workunit.client.0.smithi129.stderr:261486: start osd.9 2024-03-26T00:00:04.039 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recs.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:00:04.039 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:00:04.039 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-26T00:00:04.039 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-26T00:00:04.039 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:00:04.039 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:00:04.039 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:00:04.039 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:00:04.039 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:00:04.040 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:00:04.040 INFO:tasks.workunit.client.0.smithi129.stderr:261487: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:00:04.040 INFO:tasks.workunit.client.0.smithi129.stderr:261487: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:00:04.040 INFO:tasks.workunit.client.0.smithi129.stderr:261487: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:00:04.040 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:00:04.040 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:00:04.040 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:00:04.040 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:00:04.040 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:00:04.040 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:00:04.040 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:00:04.040 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:00:04.040 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:00:04.040 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:00:04.041 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:00:04.041 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-26T00:00:04.041 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-26T00:00:04.041 INFO:tasks.workunit.client.0.smithi129.stderr:261487: start osd.5 2024-03-26T00:00:08.816 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:00:08.816 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-03-26T00:00:08.817 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-03-26T00:00:08.817 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:00:08.817 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:00:08.817 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:00:08.817 INFO:tasks.workunit.client.0.smithi129.stderr:261486: 2024-03-26T00:00:04.075+0000 7f6bafee27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:00:08.817 INFO:tasks.workunit.client.0.smithi129.stderr:261486: 2024-03-26T00:00:04.079+0000 7f6bafee27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:00:08.817 INFO:tasks.workunit.client.0.smithi129.stderr:261486: 2024-03-26T00:00:04.083+0000 7f6bafee27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:00:08.817 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-03-26T00:00:08.817 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:00:08.817 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-26T00:00:08.817 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:00:08.817 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:00:08.817 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:00:08.817 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:00:08.818 INFO:tasks.workunit.client.0.smithi129.stderr:261486: 0 2024-03-26T00:00:08.818 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:00:08.818 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-26T00:00:08.818 INFO:tasks.workunit.client.0.smithi129.stderr:261486: 2024-03-26T00:00:04.631+0000 7f6bafee27c0 -1 Falling back to public interface 2024-03-26T00:00:08.818 INFO:tasks.workunit.client.0.smithi129.stderr:261486: 2024-03-26T00:00:04.631+0000 7f6baeea2640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6842/0: (98) Address already in use 2024-03-26T00:00:08.818 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:00:08.818 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:00:08.818 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:00:08.818 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:00:08.818 INFO:tasks.workunit.client.0.smithi129.stderr:261486: 1 2024-03-26T00:00:08.818 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:00:08.818 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-26T00:00:08.818 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:00:08.818 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:00:08.818 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:00:08.819 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:00:08.819 INFO:tasks.workunit.client.0.smithi129.stderr:261486: 2 2024-03-26T00:00:08.819 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:00:08.819 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-26T00:00:08.819 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:00:08.819 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:00:08.819 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:00:08.819 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:00:08.819 INFO:tasks.workunit.client.0.smithi129.stderr:261486: 3 2024-03-26T00:00:08.819 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:00:08.819 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-26T00:00:08.977 INFO:tasks.workunit.client.0.smithi129.stderr:26148overy_ops 2024-03-26T00:00:08.978 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-26T00:00:08.978 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-26T00:00:08.978 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:00:08.978 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:00:08.978 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:00:08.978 INFO:tasks.workunit.client.0.smithi129.stderr:261487: 2024-03-26T00:00:04.071+0000 7fce8b5587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:00:08.978 INFO:tasks.workunit.client.0.smithi129.stderr:261487: 2024-03-26T00:00:04.079+0000 7fce8b5587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:00:08.978 INFO:tasks.workunit.client.0.smithi129.stderr:261487: 2024-03-26T00:00:04.083+0000 7fce8b5587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:00:08.978 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-26T00:00:08.978 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:00:08.978 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-26T00:00:08.979 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:00:08.979 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:00:08.979 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:00:08.979 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:00:08.979 INFO:tasks.workunit.client.0.smithi129.stderr:261487: 0 2024-03-26T00:00:08.979 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:00:08.979 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-26T00:00:08.979 INFO:tasks.workunit.client.0.smithi129.stderr:261487: 2024-03-26T00:00:04.631+0000 7fce8b5587c0 -1 Falling back to public interface 2024-03-26T00:00:08.979 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:00:08.979 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:00:08.979 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:00:08.979 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:00:08.979 INFO:tasks.workunit.client.0.smithi129.stderr:261487: 1 2024-03-26T00:00:08.979 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:00:08.980 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-26T00:00:08.980 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:00:08.980 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:00:08.980 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:00:08.980 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:00:08.980 INFO:tasks.workunit.client.0.smithi129.stderr:261487: 2 2024-03-26T00:00:08.980 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:00:08.980 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-26T00:00:08.980 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:00:08.980 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:00:08.980 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:00:08.980 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:00:08.980 INFO:tasks.workunit.client.0.smithi129.stderr:261487: 3 2024-03-26T00:00:08.981 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:00:08.981 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-26T00:00:08.981 INFO:tasks.workunit.client.0.smithi129.stderr:261487: 2024-03-26T00:00:08.835+0000 7fce8b5587c0 -1 osd.5 99 log_to_monitors true 2024-03-26T00:00:11.338 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/c6: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:00:11.338 INFO:tasks.workunit.client.0.smithi129.stderr:261486: 2024-03-26T00:00:08.895+0000 7f6bafee27c0 -1 osd.9 99 log_to_monitors true 2024-03-26T00:00:11.339 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:00:11.339 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:00:11.339 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-26T00:00:11.339 INFO:tasks.workunit.client.0.smithi129.stderr:261486: 4 2024-03-26T00:00:11.339 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:00:11.339 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-26T00:00:11.339 INFO:tasks.workunit.client.0.smithi129.stderr:261486: osd.9 up in weight 1 up_from 104 up_thru 0 down_at 100 last_clean_interval [96,99) [v2:127.0.0.1:6843/2521881459,v1:127.0.0.1:6845/2521881459] [v2:127.0.0.1:6847/2521881459,v1:127.0.0.1:6849/2521881459] exists,up 5fb70f5f-d7d5-45c0-b9fd-86c8b6646430 2024-03-26T00:00:11.339 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:00:11.339 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:00:11.339 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:00:11.339 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:00:11.339 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:00:11.339 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:00:11.340 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:00:11.340 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:00:11.340 INFO:tasks.workunit.client.0.smithi129.stderr:261486: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:00:11.340 INFO:tasks.workunit.client.0.smithi129.stderr:261486: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:00:11.340 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:00:11.340 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:00:11.340 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:00:11.340 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:00:11.340 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:00:11.340 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:00:11.340 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:00:11.340 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:00:11.340 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:00:11.340 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:00:11.341 INFO:tasks.workunit.client.0.smithi129.stderr:261486: 1 2024-03-26T00:00:11.341 INFO:tasks.workunit.client.0.smithi129.stderr:261486: 2 2024-03-26T00:00:11.341 INFO:tasks.workunit.client.0.smithi129.stderr:261486: 3 2024-03-26T00:00:11.341 INFO:tasks.workunit.client.0.smithi129.stderr:261486: 4 2024-03-26T00:00:11.341 INFO:tasks.workunit.client.0.smithi129.stderr:261486: 5 2024-03-26T00:00:11.341 INFO:tasks.workunit.client.0.smithi129.stderr:261486: 6 2024-03-26T00:00:11.341 INFO:tasks.workunit.client.0.smithi129.stderr:261486: 7 2024-03-26T00:00:11.341 INFO:tasks.workunit.client.0.smithi129.stderr:261486: 8 2024-03-26T00:00:11.341 INFO:tasks.workunit.client.0.smithi129.stderr:261486: 9' 2024-03-26T00:00:11.341 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:00:11.341 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:11.341 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:00:11.342 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836592 2024-03-26T00:00:11.342 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836592 2024-03-26T00:00:11.658 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_staeph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:00:11.658 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:00:11.658 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:00:11.658 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-26T00:00:11.659 INFO:tasks.workunit.client.0.smithi129.stderr:261487: 4 2024-03-26T00:00:11.659 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:00:11.659 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-26T00:00:11.659 INFO:tasks.workunit.client.0.smithi129.stderr:261487: osd.5 up in weight 1 up_from 104 up_thru 98 down_at 100 last_clean_interval [98,99) [v2:127.0.0.1:6842/4200811718,v1:127.0.0.1:6844/4200811718] [v2:127.0.0.1:6846/4200811718,v1:127.0.0.1:6848/4200811718] exists,up 67160811-ca71-46db-8586-da42b099b746 2024-03-26T00:00:11.659 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:00:11.659 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:00:11.660 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:00:11.660 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:00:11.660 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:00:11.660 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:00:11.660 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:00:11.660 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:00:11.660 INFO:tasks.workunit.client.0.smithi129.stderr:261487: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:00:11.661 INFO:tasks.workunit.client.0.smithi129.stderr:261487: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:00:11.661 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:00:11.661 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:00:11.661 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:00:11.661 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:00:11.661 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:00:11.662 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:00:11.662 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:00:11.662 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:00:11.662 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:00:11.662 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:00:11.662 INFO:tasks.workunit.client.0.smithi129.stderr:261487: 1 2024-03-26T00:00:11.663 INFO:tasks.workunit.client.0.smithi129.stderr:261487: 2 2024-03-26T00:00:11.663 INFO:tasks.workunit.client.0.smithi129.stderr:261487: 3 2024-03-26T00:00:11.663 INFO:tasks.workunit.client.0.smithi129.stderr:261487: 4 2024-03-26T00:00:11.663 INFO:tasks.workunit.client.0.smithi129.stderr:261487: 5 2024-03-26T00:00:11.663 INFO:tasks.workunit.client.0.smithi129.stderr:261487: 6 2024-03-26T00:00:11.663 INFO:tasks.workunit.client.0.smithi129.stderr:261487: 7 2024-03-26T00:00:11.663 INFO:tasks.workunit.client.0.smithi129.stderr:261487: 8 2024-03-26T00:00:11.664 INFO:tasks.workunit.client.0.smithi129.stderr:261487: 9' 2024-03-26T00:00:11.664 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:00:11.664 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:11.664 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:00:11.664 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836593 2024-03-26T00:00:11.664 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836593 2024-03-26T00:00:11.664 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593' 2024-03-26T00:00:13.246 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ts: seqs=' 0-21474836592' 2024-03-26T00:00:13.247 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:13.247 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:00:13.247 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673061 2024-03-26T00:00:13.247 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673061 2024-03-26T00:00:13.247 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673061' 2024-03-26T00:00:13.247 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:13.247 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:00:13.247 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509532 2024-03-26T00:00:13.247 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509532 2024-03-26T00:00:13.247 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673061 2-64424509532' 2024-03-26T00:00:13.247 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:13.248 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-26T00:00:13.248 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612633 2024-03-26T00:00:13.248 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612633 2024-03-26T00:00:13.248 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673061 2-64424509532 3-313532612633' 2024-03-26T00:00:13.248 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:13.248 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-26T00:00:13.248 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182478 2024-03-26T00:00:13.248 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182478 2024-03-26T00:00:13.248 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673061 2-64424509532 3-313532612633 4-107374182478' 2024-03-26T00:00:13.248 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:13.248 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-26T00:00:13.248 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=446676598786 2024-03-26T00:00:13.248 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 446676598786 2024-03-26T00:00:13.248 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673061 2-64424509532 3-313532612633 4-107374182478 5-446676598786' 2024-03-26T00:00:13.248 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:13.248 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-26T00:00:13.249 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855420 2024-03-26T00:00:13.249 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855420 2024-03-26T00:00:13.249 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673061 2-64424509532 3-313532612633 4-107374182478 5-446676598786 6-150323855420' 2024-03-26T00:00:13.249 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:13.522 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-heids 2024-03-26T00:00:13.522 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:00:13.522 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673062 2024-03-26T00:00:13.522 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673062 2024-03-26T00:00:13.522 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673062' 2024-03-26T00:00:13.522 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:13.522 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:00:13.523 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509533 2024-03-26T00:00:13.523 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509533 2024-03-26T00:00:13.523 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673062 2-64424509533' 2024-03-26T00:00:13.523 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:13.523 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-26T00:00:13.523 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612634 2024-03-26T00:00:13.523 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612634 2024-03-26T00:00:13.523 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673062 2-64424509533 3-313532612634' 2024-03-26T00:00:13.523 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:13.523 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-26T00:00:13.523 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182479 2024-03-26T00:00:13.523 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182479 2024-03-26T00:00:13.523 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673062 2-64424509533 3-313532612634 4-107374182479' 2024-03-26T00:00:13.524 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:13.524 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-26T00:00:13.524 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=446676598787 2024-03-26T00:00:13.524 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 446676598787 2024-03-26T00:00:13.524 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673062 2-64424509533 3-313532612634 4-107374182479 5-446676598787' 2024-03-26T00:00:13.524 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:13.524 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-26T00:00:13.524 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855421 2024-03-26T00:00:13.524 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855421 2024-03-26T00:00:13.524 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673062 2-64424509533 3-313532612634 4-107374182479 5-446676598787 6-150323855421' 2024-03-26T00:00:13.524 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:13.524 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-26T00:00:13.957 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.shlpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-26T00:00:13.958 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691898 2024-03-26T00:00:13.958 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691898 2024-03-26T00:00:13.958 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673061 2-64424509532 3-313532612633 4-107374182478 5-446676598786 6-150323855420 7-171798691898' 2024-03-26T00:00:13.958 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:13.958 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-26T00:00:13.961 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528368 2024-03-26T00:00:13.961 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528368 2024-03-26T00:00:13.961 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673061 2-64424509532 3-313532612633 4-107374182478 5-446676598786 6-150323855420 7-171798691898 8-193273528368' 2024-03-26T00:00:13.961 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:13.962 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-26T00:00:13.962 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=446676598786 2024-03-26T00:00:13.962 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 446676598786 2024-03-26T00:00:13.962 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673061 2-64424509532 3-313532612633 4-107374182478 5-446676598786 6-150323855420 7-171798691898 8-193273528368 9-446676598786' 2024-03-26T00:00:13.962 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:13.962 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836592 2024-03-26T00:00:13.962 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:13.962 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:00:13.962 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836592 2024-03-26T00:00:13.962 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:13.962 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836592 2024-03-26T00:00:13.962 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836592' 2024-03-26T00:00:13.963 INFO:tasks.workunit.client.0.smithi129.stderr:261486: waiting osd.0 seq 21474836592 2024-03-26T00:00:13.963 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:00:13.963 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836593 -lt 21474836592 2024-03-26T00:00:13.963 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:13.963 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673061 2024-03-26T00:00:13.963 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:13.963 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:00:13.963 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673061 2024-03-26T00:00:13.963 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:13.963 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673061 2024-03-26T00:00:14.118 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd:2202: flush_pg_stats: seq=171798691899 2024-03-26T00:00:14.118 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691899 2024-03-26T00:00:14.118 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673062 2-64424509533 3-313532612634 4-107374182479 5-446676598787 6-150323855421 7-171798691899' 2024-03-26T00:00:14.118 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:14.118 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-26T00:00:14.118 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528369 2024-03-26T00:00:14.118 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528369 2024-03-26T00:00:14.118 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673062 2-64424509533 3-313532612634 4-107374182479 5-446676598787 6-150323855421 7-171798691899 8-193273528369' 2024-03-26T00:00:14.118 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:14.118 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-26T00:00:14.119 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=446676598787 2024-03-26T00:00:14.119 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 446676598787 2024-03-26T00:00:14.119 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673062 2-64424509533 3-313532612634 4-107374182479 5-446676598787 6-150323855421 7-171798691899 8-193273528369 9-446676598787' 2024-03-26T00:00:14.119 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:14.119 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836593 2024-03-26T00:00:14.119 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:14.119 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:00:14.119 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836593 2024-03-26T00:00:14.119 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:14.119 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836593 2024-03-26T00:00:14.119 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836593' 2024-03-26T00:00:14.119 INFO:tasks.workunit.client.0.smithi129.stderr:261487: waiting osd.0 seq 21474836593 2024-03-26T00:00:14.119 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:00:14.120 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836593 -lt 21474836593 2024-03-26T00:00:14.120 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:14.120 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673062 2024-03-26T00:00:14.120 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:14.120 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:00:14.120 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673062 2024-03-26T00:00:14.120 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:14.120 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673062 2024-03-26T00:00:14.120 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673062' 2024-03-26T00:00:14.120 INFO:tasks.workunit.client.0.smithi129.stderr:261487: waiting osd.1 seq 42949673062 2024-03-26T00:00:15.305 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:221.1 seq 42949673061' 2024-03-26T00:00:15.305 INFO:tasks.workunit.client.0.smithi129.stderr:261486: waiting osd.1 seq 42949673061 2024-03-26T00:00:15.305 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:00:15.305 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673062 -lt 42949673061 2024-03-26T00:00:15.305 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:15.305 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:15.305 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509532 2024-03-26T00:00:15.305 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:00:15.305 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509532 2024-03-26T00:00:15.306 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:15.306 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509532 2024-03-26T00:00:15.306 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509532' 2024-03-26T00:00:15.306 INFO:tasks.workunit.client.0.smithi129.stderr:261486: waiting osd.2 seq 64424509532 2024-03-26T00:00:15.306 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:00:15.306 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509533 -lt 64424509532 2024-03-26T00:00:15.306 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:15.306 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-313532612633 2024-03-26T00:00:15.306 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:15.306 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-26T00:00:15.306 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-313532612633 2024-03-26T00:00:15.306 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:15.306 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612633 2024-03-26T00:00:15.306 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 313532612633' 2024-03-26T00:00:15.307 INFO:tasks.workunit.client.0.smithi129.stderr:261486: waiting osd.3 seq 313532612633 2024-03-26T00:00:15.307 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-26T00:00:15.307 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612634 -lt 313532612633 2024-03-26T00:00:15.307 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:15.307 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182478 2024-03-26T00:00:15.307 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:15.307 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-26T00:00:15.307 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182478 2024-03-26T00:00:15.307 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:15.307 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182478 2024-03-26T00:00:15.307 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182478' 2024-03-26T00:00:15.308 INFO:tasks.workunit.client.0.smithi129.stderr:261486: waiting osd.4 seq 107374182478 2024-03-26T00:00:15.308 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-26T00:00:15.308 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182480 -lt 107374182478 2024-03-26T00:00:15.399 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in4: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:00:15.400 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673062 -lt 42949673062 2024-03-26T00:00:15.400 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:15.400 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509533 2024-03-26T00:00:15.400 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:15.400 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:00:15.400 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509533 2024-03-26T00:00:15.400 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:15.400 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509533 2024-03-26T00:00:15.400 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509533' 2024-03-26T00:00:15.400 INFO:tasks.workunit.client.0.smithi129.stderr:261487: waiting osd.2 seq 64424509533 2024-03-26T00:00:15.400 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:00:15.400 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509533 -lt 64424509533 2024-03-26T00:00:15.400 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:15.401 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-313532612634 2024-03-26T00:00:15.401 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:15.401 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-26T00:00:15.401 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-313532612634 2024-03-26T00:00:15.401 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:15.401 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612634 2024-03-26T00:00:15.401 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 313532612634' 2024-03-26T00:00:15.401 INFO:tasks.workunit.client.0.smithi129.stderr:261487: waiting osd.3 seq 313532612634 2024-03-26T00:00:15.401 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-26T00:00:15.401 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612634 -lt 313532612634 2024-03-26T00:00:15.401 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:15.401 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182479 2024-03-26T00:00:15.401 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:15.401 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-26T00:00:15.402 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182479 2024-03-26T00:00:15.402 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:15.402 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182479 2024-03-26T00:00:15.402 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182479' 2024-03-26T00:00:15.402 INFO:tasks.workunit.client.0.smithi129.stderr:261487: waiting osd.4 seq 107374182479 2024-03-26T00:00:15.402 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-26T00:00:15.402 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182480 -lt 107374182479 2024-03-26T00:00:15.402 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:15.402 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-446676598787 2024-03-26T00:00:16.327 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:16.327 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-26T00:00:16.327 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-446676598787 2024-03-26T00:00:16.327 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:16.327 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=446676598787 2024-03-26T00:00:16.327 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 446676598787' 2024-03-26T00:00:16.327 INFO:tasks.workunit.client.0.smithi129.stderr:261487: waiting osd.5 seq 446676598787 2024-03-26T00:00:16.327 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-26T00:00:16.327 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 446676598787 -lt 446676598787 2024-03-26T00:00:16.328 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:16.328 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855421 2024-03-26T00:00:16.328 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:16.328 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-26T00:00:16.328 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855421 2024-03-26T00:00:16.328 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:16.328 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855421 2024-03-26T00:00:16.328 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855421' 2024-03-26T00:00:16.328 INFO:tasks.workunit.client.0.smithi129.stderr:261487: waiting osd.6 seq 150323855421 2024-03-26T00:00:16.328 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-26T00:00:16.328 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855422 -lt 150323855421 2024-03-26T00:00:16.328 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:16.328 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:16.328 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691899 2024-03-26T00:00:16.329 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-26T00:00:16.329 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691899 2024-03-26T00:00:16.329 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:16.329 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691899 2024-03-26T00:00:16.329 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691899' 2024-03-26T00:00:16.329 INFO:tasks.workunit.client.0.smithi129.stderr:261487: waiting osd.7 seq 171798691899 2024-03-26T00:00:16.329 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-26T00:00:16.329 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691900 -lt 171798691899 2024-03-26T00:00:16.329 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:16.329 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528369 2024-03-26T00:00:16.329 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:16.329 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-26T00:00:16.329 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528369 2024-03-26T00:00:16.337 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush $seqs 2024-03-26T00:00:16.337 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-446676598786 2024-03-26T00:00:16.337 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:16.338 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-26T00:00:16.338 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-446676598786 2024-03-26T00:00:16.338 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:16.338 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=446676598786 2024-03-26T00:00:16.338 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 446676598786' 2024-03-26T00:00:16.338 INFO:tasks.workunit.client.0.smithi129.stderr:261486: waiting osd.5 seq 446676598786 2024-03-26T00:00:16.338 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-26T00:00:16.338 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 446676598787 -lt 446676598786 2024-03-26T00:00:16.338 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:16.338 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855420 2024-03-26T00:00:16.338 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:16.338 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-26T00:00:16.338 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855420 2024-03-26T00:00:16.339 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:16.339 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855420 2024-03-26T00:00:16.339 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855420' 2024-03-26T00:00:16.339 INFO:tasks.workunit.client.0.smithi129.stderr:261486: waiting osd.6 seq 150323855420 2024-03-26T00:00:16.339 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-26T00:00:16.339 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855422 -lt 150323855420 2024-03-26T00:00:16.339 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:16.339 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691898 2024-03-26T00:00:16.339 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:16.339 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-26T00:00:16.339 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691898 2024-03-26T00:00:16.339 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:16.339 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691898 2024-03-26T00:00:16.340 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691898' 2024-03-26T00:00:16.340 INFO:tasks.workunit.client.0.smithi129.stderr:261486: waiting osd.7 seq 171798691898 2024-03-26T00:00:16.340 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-26T00:00:16.340 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691900 -lt 171798691898 2024-03-26T00:00:16.340 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:16.340 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528368 2024-03-26T00:00:16.340 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:16.340 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-26T00:00:18.078 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528368 2024-03-26T00:00:18.078 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:18.078 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528368 2024-03-26T00:00:18.078 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528368' 2024-03-26T00:00:18.079 INFO:tasks.workunit.client.0.smithi129.stderr:261486: waiting osd.8 seq 193273528368 2024-03-26T00:00:18.079 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-26T00:00:18.079 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528370 -lt 193273528368 2024-03-26T00:00:18.079 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:18.079 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-446676598786 2024-03-26T00:00:18.079 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:18.079 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-26T00:00:18.079 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-446676598786 2024-03-26T00:00:18.079 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:18.080 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=446676598786 2024-03-26T00:00:18.080 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 446676598786' 2024-03-26T00:00:18.080 INFO:tasks.workunit.client.0.smithi129.stderr:261486: waiting osd.9 seq 446676598786 2024-03-26T00:00:18.080 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-26T00:00:18.080 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 446676598788 -lt 446676598786 2024-03-26T00:00:18.080 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:00:18.080 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:00:18.080 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:00:18.080 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-26T00:00:18.080 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:00:18.080 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:00:18.080 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:00:18.081 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:00:18.081 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:00:18.081 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:00:18.081 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:00:18.081 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-26T00:00:18.081 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:00:18.081 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:00:18.081 INFO:tasks.workunit.client.0.smithi129.stderr:261486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:00:18.081 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-26T00:00:18.081 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:00:18.081 INFO:tasks.workunit.client.0.smithi129.stderr:261486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:00:18.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-26T00:00:18.087 INFO:tasks.workunit.client.0.smithi129.stderr:_pg_stats: cut -d - -f 2 2024-03-26T00:00:18.087 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528369 2024-03-26T00:00:18.087 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528369' 2024-03-26T00:00:18.088 INFO:tasks.workunit.client.0.smithi129.stderr:261487: waiting osd.8 seq 193273528369 2024-03-26T00:00:18.088 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-26T00:00:18.088 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528370 -lt 193273528369 2024-03-26T00:00:18.088 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:18.088 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-446676598787 2024-03-26T00:00:18.088 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:18.088 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-26T00:00:18.088 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-446676598787 2024-03-26T00:00:18.088 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:18.088 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=446676598787 2024-03-26T00:00:18.088 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 446676598787' 2024-03-26T00:00:18.089 INFO:tasks.workunit.client.0.smithi129.stderr:261487: waiting osd.9 seq 446676598787 2024-03-26T00:00:18.089 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-26T00:00:18.089 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 446676598788 -lt 446676598787 2024-03-26T00:00:18.089 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:00:18.089 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:00:18.089 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:00:18.089 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-26T00:00:18.089 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:00:18.089 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:00:18.089 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:00:18.089 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:00:18.089 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:00:18.089 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:00:18.090 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:00:18.090 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-26T00:00:18.090 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:00:18.090 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:00:18.090 INFO:tasks.workunit.client.0.smithi129.stderr:261487: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:00:18.090 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-26T00:00:18.090 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:00:18.090 INFO:tasks.workunit.client.0.smithi129.stderr:261487: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:00:18.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-26T00:00:18.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-26T00:00:18.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 261483 2024-03-26T00:00:18.091 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-26T00:00:18.091 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-26T00:00:18.091 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-26T00:00:18.091 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-03-26T00:00:18.091 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-26T00:00:18.091 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-26T00:00:18.212 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-26T00:00:18.216 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2024-03-26T00:00:18.216 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-03-26T00:00:18.216 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-03-26T00:00:18.216 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-03-26T00:00:18.216 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=5 2024-03-26T00:00:18.216 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-03-26T00:00:18.216 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-26T00:00:18.216 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-26T00:00:18.216 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-26T00:00:18.216 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 264464"' 2024-03-26T00:00:18.217 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 264464' 2024-03-26T00:00:18.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-26T00:00:18.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-26T00:00:18.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-26T00:00:18.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-26T00:00:18.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 264467"' 2024-03-26T00:00:18.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/264466: /' 2024-03-26T00:00:18.217 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 264467' 2024-03-26T00:00:18.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-03-26T00:00:18.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 264464 264467' 2024-03-26T00:00:18.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-26T00:00:18.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-26T00:00:18.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 264464 2024-03-26T00:00:18.218 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-26T00:00:18.218 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/264469: /' 2024-03-26T00:00:21.865 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:00:21.866 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:00:21.866 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-26T00:00:21.866 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:00:21.866 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-03-26T00:00:21.866 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:00:21.866 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:00:21.866 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-26T00:00:21.866 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:00:21.866 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-26T00:00:21.866 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:00:21.866 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:00:21.867 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:00:21.867 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:00:21.867 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:00:21.867 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:00:21.867 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-03-26T00:00:21.867 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:00:21.867 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:00:21.867 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-26T00:00:21.867 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:00:21.867 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-26T00:00:21.867 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-03-26T00:00:21.867 INFO:tasks.workunit.client.0.smithi129.stderr:264466: remove 0#2:eb822e21:::SOMETHING:head# 2024-03-26T00:00:21.868 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-26T00:00:21.868 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:00:21.868 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:00:21.868 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-26T00:00:21.868 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:00:21.868 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-26T00:00:21.868 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:00:21.868 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:00:21.870 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:00:21.870 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:00:21.870 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-26T00:00:21.870 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-26T00:00:21.870 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:00:21.871 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:00:21.871 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:00:21.871 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:00:21.871 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:00:21.871 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:00:21.871 INFO:tasks.workunit.client.0.smithi129.stderr:264466: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:00:21.871 INFO:tasks.workunit.client.0.smithi129.stderr:264466: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:00:21.871 INFO:tasks.workunit.client.0.smithi129.stderr:264466: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:00:21.871 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:00:21.871 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:00:21.871 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:00:21.871 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:00:21.871 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:00:21.872 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:00:21.872 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:00:21.872 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:00:21.872 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:00:21.872 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:00:21.872 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:00:21.872 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-26T00:00:21.872 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-26T00:00:21.872 INFO:tasks.workunit.client.0.smithi129.stderr:264466: start osd.3 2024-03-26T00:00:21.872 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:00:22.636 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:00:22.636 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:00:22.636 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-26T00:00:22.636 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:00:22.636 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-03-26T00:00:22.636 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:00:22.636 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:00:22.636 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-26T00:00:22.636 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:00:22.637 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-26T00:00:22.637 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:00:22.637 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:00:22.637 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:00:22.637 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:00:22.637 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:00:22.637 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:00:22.637 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-03-26T00:00:22.637 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:00:22.637 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:00:22.637 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-26T00:00:22.638 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:00:22.638 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-26T00:00:22.638 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-03-26T00:00:22.638 INFO:tasks.workunit.client.0.smithi129.stderr:264469: remove 1#2:eb822e21:::SOMETHING:head# 2024-03-26T00:00:22.638 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-26T00:00:22.638 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:00:22.638 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:00:22.638 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-26T00:00:22.638 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:00:22.639 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-26T00:00:22.639 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:00:22.639 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:00:22.639 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:00:22.640 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:00:22.640 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-26T00:00:22.640 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-26T00:00:22.640 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:00:22.640 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:00:22.640 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:00:22.640 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:00:22.640 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:00:22.640 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:00:22.640 INFO:tasks.workunit.client.0.smithi129.stderr:264469: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:00:22.641 INFO:tasks.workunit.client.0.smithi129.stderr:264469: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:00:22.641 INFO:tasks.workunit.client.0.smithi129.stderr:264469: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:00:22.641 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:00:22.641 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:00:22.641 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:00:22.641 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:00:22.641 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:00:22.641 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:00:22.641 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:00:22.641 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:00:22.641 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:00:22.642 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:00:22.642 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:00:22.642 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-26T00:00:22.642 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-26T00:00:22.642 INFO:tasks.workunit.client.0.smithi129.stderr:264469: start osd.5 2024-03-26T00:00:22.642 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:00:26.766 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-26T00:00:26.766 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-26T00:00:26.767 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:00:26.767 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:00:26.767 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:00:26.767 INFO:tasks.workunit.client.0.smithi129.stderr:264466: 2024-03-26T00:00:21.902+0000 7f6e823eb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:00:26.767 INFO:tasks.workunit.client.0.smithi129.stderr:264466: 2024-03-26T00:00:21.926+0000 7f6e823eb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:00:26.767 INFO:tasks.workunit.client.0.smithi129.stderr:264466: 2024-03-26T00:00:21.926+0000 7f6e823eb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:00:26.767 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-26T00:00:26.767 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:00:26.767 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-26T00:00:26.767 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:00:26.767 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:00:26.767 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:00:26.767 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:00:26.768 INFO:tasks.workunit.client.0.smithi129.stderr:264466: 0 2024-03-26T00:00:26.768 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:00:26.768 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-26T00:00:26.768 INFO:tasks.workunit.client.0.smithi129.stderr:264466: 2024-03-26T00:00:22.482+0000 7f6e823eb7c0 -1 Falling back to public interface 2024-03-26T00:00:26.768 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:00:26.768 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:00:26.768 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:00:26.768 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:00:26.768 INFO:tasks.workunit.client.0.smithi129.stderr:264466: 1 2024-03-26T00:00:26.768 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-26T00:00:26.768 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:00:26.768 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:00:26.768 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:00:26.768 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:00:26.769 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:00:26.769 INFO:tasks.workunit.client.0.smithi129.stderr:264466: 2 2024-03-26T00:00:26.769 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:00:26.769 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-26T00:00:26.769 INFO:tasks.workunit.client.0.smithi129.stderr:264466: 2024-03-26T00:00:25.178+0000 7f6e823eb7c0 -1 osd.3 107 log_to_monitors true 2024-03-26T00:00:26.769 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:00:26.769 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:00:26.769 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:00:26.769 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:00:26.769 INFO:tasks.workunit.client.0.smithi129.stderr:264466: 3 2024-03-26T00:00:26.769 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:00:26.770 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-26T00:00:27.455 INFO:tasks.workunit.client.0.smithi129.stderr:264466: osd.3 up in weight 1 up_from 112 up_thru 112 down_at 108 l264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-26T00:00:27.455 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-26T00:00:27.455 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:00:27.455 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:00:27.455 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:00:27.455 INFO:tasks.workunit.client.0.smithi129.stderr:264469: 2024-03-26T00:00:22.670+0000 7f7f8b2b07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:00:27.455 INFO:tasks.workunit.client.0.smithi129.stderr:264469: 2024-03-26T00:00:22.698+0000 7f7f8b2b07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:00:27.456 INFO:tasks.workunit.client.0.smithi129.stderr:264469: 2024-03-26T00:00:22.702+0000 7f7f8b2b07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:00:27.456 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-26T00:00:27.456 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:00:27.456 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-26T00:00:27.456 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:00:27.456 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:00:27.456 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:00:27.456 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:00:27.456 INFO:tasks.workunit.client.0.smithi129.stderr:264469: 0 2024-03-26T00:00:27.456 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:00:27.456 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-26T00:00:27.456 INFO:tasks.workunit.client.0.smithi129.stderr:264469: 2024-03-26T00:00:23.242+0000 7f7f8b2b07c0 -1 Falling back to public interface 2024-03-26T00:00:27.456 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:00:27.456 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:00:27.457 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:00:27.457 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:00:27.457 INFO:tasks.workunit.client.0.smithi129.stderr:264469: 1 2024-03-26T00:00:27.457 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:00:27.457 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-26T00:00:27.457 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:00:27.457 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:00:27.457 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:00:27.457 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:00:27.457 INFO:tasks.workunit.client.0.smithi129.stderr:264469: 2 2024-03-26T00:00:27.457 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:00:27.457 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-26T00:00:27.457 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:00:27.457 INFO:tasks.workunit.client.0.smithi129.stderr:264469: 2024-03-26T00:00:26.914+0000 7f7f8b2b07c0 -1 osd.5 107 log_to_monitors true 2024-03-26T00:00:27.457 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:00:27.457 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:00:27.458 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:00:27.458 INFO:tasks.workunit.client.0.smithi129.stderr:264469: 3 2024-03-26T00:00:27.458 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:00:27.458 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-26T00:00:28.059 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpeast_clean_interval [73,107) [v2:127.0.0.1:6826/907779681,v1:127.0.0.1:6827/907779681] [v2:127.0.0.1:6828/907779681,v1:127.0.0.1:6829/907779681] exists,up 25e4ca68-13d2-4df8-9930-c8c629bba9ef 2024-03-26T00:00:28.059 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:00:28.059 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:00:28.060 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:00:28.060 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:00:28.060 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:00:28.060 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:00:28.060 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:00:28.060 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:00:28.060 INFO:tasks.workunit.client.0.smithi129.stderr:264466: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:00:28.060 INFO:tasks.workunit.client.0.smithi129.stderr:264466: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:00:28.060 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:00:28.060 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:00:28.060 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:00:28.061 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:00:28.061 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:00:28.061 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:00:28.061 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:00:28.061 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:00:28.061 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:00:28.061 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:00:28.061 INFO:tasks.workunit.client.0.smithi129.stderr:264466: 1 2024-03-26T00:00:28.061 INFO:tasks.workunit.client.0.smithi129.stderr:264466: 2 2024-03-26T00:00:28.061 INFO:tasks.workunit.client.0.smithi129.stderr:264466: 3 2024-03-26T00:00:28.061 INFO:tasks.workunit.client.0.smithi129.stderr:264466: 4 2024-03-26T00:00:28.061 INFO:tasks.workunit.client.0.smithi129.stderr:264466: 5 2024-03-26T00:00:28.062 INFO:tasks.workunit.client.0.smithi129.stderr:264466: 6 2024-03-26T00:00:28.062 INFO:tasks.workunit.client.0.smithi129.stderr:264466: 7 2024-03-26T00:00:28.062 INFO:tasks.workunit.client.0.smithi129.stderr:264466: 8 2024-03-26T00:00:28.062 INFO:tasks.workunit.client.0.smithi129.stderr:264466: 9' 2024-03-26T00:00:28.062 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:00:28.062 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:28.062 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:00:28.062 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836597 2024-03-26T00:00:28.062 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836597 2024-03-26T00:00:28.062 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836597' 2024-03-26T00:00:28.063 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:28.063 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:00:28.063 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673066 2024-03-26T00:00:28.063 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673066 2024-03-26T00:00:28.063 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836597 1-42949673066' 2024-03-26T00:00:28.063 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:29.810 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/rs.sh:985: wait_for_osd: sleep 1 2024-03-26T00:00:29.811 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:00:29.811 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:00:29.811 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-26T00:00:29.811 INFO:tasks.workunit.client.0.smithi129.stderr:264469: 4 2024-03-26T00:00:29.811 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:00:29.811 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-26T00:00:29.811 INFO:tasks.workunit.client.0.smithi129.stderr:264469: osd.5 up in weight 1 up_from 115 up_thru 115 down_at 108 last_clean_interval [104,107) [v2:127.0.0.1:6842/2606262139,v1:127.0.0.1:6844/2606262139] [v2:127.0.0.1:6846/2606262139,v1:127.0.0.1:6848/2606262139] exists,up 67160811-ca71-46db-8586-da42b099b746 2024-03-26T00:00:29.811 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:00:29.811 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:00:29.811 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:00:29.812 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:00:29.812 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:00:29.812 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:00:29.812 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:00:29.812 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:00:29.812 INFO:tasks.workunit.client.0.smithi129.stderr:264469: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:00:29.812 INFO:tasks.workunit.client.0.smithi129.stderr:264469: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:00:29.812 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:00:29.812 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:00:29.812 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:00:29.812 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:00:29.812 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:00:29.812 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:00:29.812 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:00:29.813 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:00:29.813 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:00:29.813 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:00:29.813 INFO:tasks.workunit.client.0.smithi129.stderr:264469: 1 2024-03-26T00:00:29.813 INFO:tasks.workunit.client.0.smithi129.stderr:264469: 2 2024-03-26T00:00:29.813 INFO:tasks.workunit.client.0.smithi129.stderr:264469: 3 2024-03-26T00:00:29.813 INFO:tasks.workunit.client.0.smithi129.stderr:264469: 4 2024-03-26T00:00:29.813 INFO:tasks.workunit.client.0.smithi129.stderr:264469: 5 2024-03-26T00:00:29.813 INFO:tasks.workunit.client.0.smithi129.stderr:264469: 6 2024-03-26T00:00:29.813 INFO:tasks.workunit.client.0.smithi129.stderr:264469: 7 2024-03-26T00:00:29.813 INFO:tasks.workunit.client.0.smithi129.stderr:264469: 8 2024-03-26T00:00:29.813 INFO:tasks.workunit.client.0.smithi129.stderr:264469: 9' 2024-03-26T00:00:29.813 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:00:29.813 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:29.814 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:00:29.814 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836599 2024-03-26T00:00:29.814 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836599 2024-03-26T00:00:29.814 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836599' 2024-03-26T00:00:29.814 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:30.280 INFO:tasks.workunit.client.0.smithi129.stderr:26qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:00:30.280 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509537 2024-03-26T00:00:30.280 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509537 2024-03-26T00:00:30.280 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836597 1-42949673066 2-64424509537' 2024-03-26T00:00:30.280 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:30.280 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-26T00:00:30.281 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=481036337154 2024-03-26T00:00:30.281 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 481036337154 2024-03-26T00:00:30.281 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836597 1-42949673066 2-64424509537 3-481036337154' 2024-03-26T00:00:30.281 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:30.281 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-26T00:00:30.281 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182484 2024-03-26T00:00:30.281 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182484 2024-03-26T00:00:30.281 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836597 1-42949673066 2-64424509537 3-481036337154 4-107374182484' 2024-03-26T00:00:30.281 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:30.281 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-26T00:00:30.281 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=493921239042 2024-03-26T00:00:30.281 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 493921239042 2024-03-26T00:00:30.281 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836597 1-42949673066 2-64424509537 3-481036337154 4-107374182484 5-493921239042' 2024-03-26T00:00:30.281 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:30.281 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-26T00:00:30.282 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855425 2024-03-26T00:00:30.282 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855425 2024-03-26T00:00:30.282 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836597 1-42949673066 2-64424509537 3-481036337154 4-107374182484 5-493921239042 6-150323855425' 2024-03-26T00:00:30.282 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:30.282 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-26T00:00:30.282 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691904 2024-03-26T00:00:30.282 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691904 2024-03-26T00:00:30.282 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836597 1-42949673066 2-64424509537 3-481036337154 4-107374182484 5-493921239042 6-150323855425 7-171798691904' 2024-03-26T00:00:30.282 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:30.282 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-26T00:00:31.149 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528374 2024-03-26T00:00:31.149 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528374 2024-03-26T00:00:31.149 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836597 1-42949673066 2-64424509537 3-481036337154 4-107374182484 5-493921239042 6-150323855425 7-171798691904 8-193273528374' 2024-03-26T00:00:31.149 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:31.149 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-26T00:00:31.149 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=446676598792 2024-03-26T00:00:31.149 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 446676598792 2024-03-26T00:00:31.149 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836597 1-42949673066 2-64424509537 3-481036337154 4-107374182484 5-493921239042 6-150323855425 7-171798691904 8-193273528374 9-446676598792' 2024-03-26T00:00:31.150 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:31.150 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836597 2024-03-26T00:00:31.150 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:31.150 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:00:31.150 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836597 2024-03-26T00:00:31.150 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:31.150 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836597 2024-03-26T00:00:31.150 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836597' 2024-03-26T00:00:31.150 INFO:tasks.workunit.client.0.smithi129.stderr:264466: waiting osd.0 seq 21474836597 2024-03-26T00:00:31.150 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:00:31.150 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836599 -lt 21474836597 2024-03-26T00:00:31.150 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:31.150 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673066 2024-03-26T00:00:31.150 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:31.151 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:00:31.151 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673066 2024-03-26T00:00:31.151 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:31.151 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673066 2024-03-26T00:00:31.151 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673066' 2024-03-26T00:00:31.151 INFO:tasks.workunit.client.0.smithi129.stderr:264466: waiting osd.1 seq 42949673066 2024-03-26T00:00:31.151 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:00:31.151 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673066 -lt 42949673066 2024-03-26T00:00:31.151 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:31.151 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509537 2024-03-26T00:00:31.151 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:31.151 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:00:31.165 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone4469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:00:31.165 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673067 2024-03-26T00:00:31.165 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673067 2024-03-26T00:00:31.165 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836599 1-42949673067' 2024-03-26T00:00:31.166 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:31.166 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:00:31.166 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509539 2024-03-26T00:00:31.166 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509539 2024-03-26T00:00:31.166 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836599 1-42949673067 2-64424509539' 2024-03-26T00:00:31.166 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:31.166 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-26T00:00:31.166 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=481036337156 2024-03-26T00:00:31.166 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 481036337156 2024-03-26T00:00:31.166 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836599 1-42949673067 2-64424509539 3-481036337156' 2024-03-26T00:00:31.166 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:31.166 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-26T00:00:31.166 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182485 2024-03-26T00:00:31.167 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182485 2024-03-26T00:00:31.167 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836599 1-42949673067 2-64424509539 3-481036337156 4-107374182485' 2024-03-26T00:00:31.167 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:31.167 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-26T00:00:31.167 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=493921239043 2024-03-26T00:00:31.167 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 493921239043 2024-03-26T00:00:31.167 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836599 1-42949673067 2-64424509539 3-481036337156 4-107374182485 5-493921239043' 2024-03-26T00:00:31.167 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:31.167 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-26T00:00:31.167 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855427 2024-03-26T00:00:31.167 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855427 2024-03-26T00:00:31.167 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836599 1-42949673067 2-64424509539 3-481036337156 4-107374182485 5-493921239043 6-150323855427' 2024-03-26T00:00:31.167 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:31.168 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-26T00:00:31.741 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691905 2024-03-26T00:00:31.741 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691905 2024-03-26T00:00:31.741 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836599 1-42949673067 2-64424509539 3-481036337156 4-107374182485 5-493921239043 6-150323855427 7-171798691905' 2024-03-26T00:00:31.741 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:31.741 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-26T00:00:31.742 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528375 2024-03-26T00:00:31.742 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528375 2024-03-26T00:00:31.742 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836599 1-42949673067 2-64424509539 3-481036337156 4-107374182485 5-493921239043 6-150323855427 7-171798691905 8-193273528375' 2024-03-26T00:00:31.742 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:31.742 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-26T00:00:31.742 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=446676598793 2024-03-26T00:00:31.745 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 446676598793 2024-03-26T00:00:31.745 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836599 1-42949673067 2-64424509539 3-481036337156 4-107374182485 5-493921239043 6-150323855427 7-171798691905 8-193273528375 9-446676598793' 2024-03-26T00:00:31.745 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:31.745 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836599 2024-03-26T00:00:31.745 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:31.745 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:00:31.746 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836599 2024-03-26T00:00:31.746 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:31.746 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836599 2024-03-26T00:00:31.746 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836599' 2024-03-26T00:00:31.746 INFO:tasks.workunit.client.0.smithi129.stderr:264469: waiting osd.0 seq 21474836599 2024-03-26T00:00:31.746 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:00:31.746 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836599 -lt 21474836599 2024-03-26T00:00:31.746 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:31.746 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673067 2024-03-26T00:00:31.746 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:31.746 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:00:31.746 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673067 2024-03-26T00:00:31.746 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:31.747 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673067 2024-03-26T00:00:31.747 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673067' 2024-03-26T00:00:31.747 INFO:tasks.workunit.client.0.smithi129.stderr:264469: waiting osd.1 seq 42949673067 2024-03-26T00:00:32.161 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flu.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509537 2024-03-26T00:00:32.161 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:32.161 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509537 2024-03-26T00:00:32.161 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509537' 2024-03-26T00:00:32.161 INFO:tasks.workunit.client.0.smithi129.stderr:264466: waiting osd.2 seq 64424509537 2024-03-26T00:00:32.161 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:00:32.161 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509538 -lt 64424509537 2024-03-26T00:00:32.161 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:32.161 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-481036337154 2024-03-26T00:00:32.161 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:32.161 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-26T00:00:32.162 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-481036337154 2024-03-26T00:00:32.162 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:32.162 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=481036337154 2024-03-26T00:00:32.162 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 481036337154' 2024-03-26T00:00:32.162 INFO:tasks.workunit.client.0.smithi129.stderr:264466: waiting osd.3 seq 481036337154 2024-03-26T00:00:32.162 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-26T00:00:32.162 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 481036337154 -lt 481036337154 2024-03-26T00:00:32.162 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:32.162 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182484 2024-03-26T00:00:32.162 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:32.162 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-26T00:00:32.162 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182484 2024-03-26T00:00:32.162 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:32.162 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182484 2024-03-26T00:00:32.163 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182484' 2024-03-26T00:00:32.163 INFO:tasks.workunit.client.0.smithi129.stderr:264466: waiting osd.4 seq 107374182484 2024-03-26T00:00:32.163 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-26T00:00:32.163 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182485 -lt 107374182484 2024-03-26T00:00:32.163 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:32.163 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-493921239042 2024-03-26T00:00:32.163 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:32.163 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-26T00:00:32.163 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-493921239042 2024-03-26T00:00:32.163 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:32.163 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=493921239042 2024-03-26T00:00:33.098 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waish_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:00:33.099 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673068 -lt 42949673067 2024-03-26T00:00:33.099 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:33.099 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509539 2024-03-26T00:00:33.099 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:33.099 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:00:33.099 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509539 2024-03-26T00:00:33.099 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:33.099 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509539 2024-03-26T00:00:33.099 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509539' 2024-03-26T00:00:33.099 INFO:tasks.workunit.client.0.smithi129.stderr:264469: waiting osd.2 seq 64424509539 2024-03-26T00:00:33.099 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:00:33.100 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509539 -lt 64424509539 2024-03-26T00:00:33.100 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:33.100 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-481036337156 2024-03-26T00:00:33.100 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:33.100 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-26T00:00:33.100 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-481036337156 2024-03-26T00:00:33.100 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:33.100 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=481036337156 2024-03-26T00:00:33.100 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 481036337156' 2024-03-26T00:00:33.100 INFO:tasks.workunit.client.0.smithi129.stderr:264469: waiting osd.3 seq 481036337156 2024-03-26T00:00:33.100 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-26T00:00:33.101 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 481036337156 -lt 481036337156 2024-03-26T00:00:33.101 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:33.101 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182485 2024-03-26T00:00:33.101 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:33.101 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-26T00:00:33.101 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182485 2024-03-26T00:00:33.101 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:33.101 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182485 2024-03-26T00:00:33.101 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182485' 2024-03-26T00:00:33.101 INFO:tasks.workunit.client.0.smithi129.stderr:264469: waiting osd.4 seq 107374182485 2024-03-26T00:00:33.101 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-26T00:00:33.102 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182485 -lt 107374182485 2024-03-26T00:00:33.102 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:33.102 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-493921239043 2024-03-26T00:00:33.426 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubting osd.5 seq 493921239042' 2024-03-26T00:00:33.426 INFO:tasks.workunit.client.0.smithi129.stderr:264466: waiting osd.5 seq 493921239042 2024-03-26T00:00:33.426 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-26T00:00:33.426 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 493921239043 -lt 493921239042 2024-03-26T00:00:33.426 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:33.426 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855425 2024-03-26T00:00:33.426 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:33.426 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-26T00:00:33.426 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855425 2024-03-26T00:00:33.426 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:33.427 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855425 2024-03-26T00:00:33.427 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855425' 2024-03-26T00:00:33.427 INFO:tasks.workunit.client.0.smithi129.stderr:264466: waiting osd.6 seq 150323855425 2024-03-26T00:00:33.427 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-26T00:00:33.427 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855427 -lt 150323855425 2024-03-26T00:00:33.427 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:33.427 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691904 2024-03-26T00:00:33.427 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:33.427 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-26T00:00:33.427 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691904 2024-03-26T00:00:33.427 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:33.427 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691904 2024-03-26T00:00:33.428 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691904' 2024-03-26T00:00:33.428 INFO:tasks.workunit.client.0.smithi129.stderr:264466: waiting osd.7 seq 171798691904 2024-03-26T00:00:33.428 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-26T00:00:33.428 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691905 -lt 171798691904 2024-03-26T00:00:33.428 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:33.428 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528374 2024-03-26T00:00:33.428 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:33.428 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-26T00:00:33.428 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528374 2024-03-26T00:00:33.428 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:33.428 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528374 2024-03-26T00:00:33.428 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528374' 2024-03-26T00:00:33.429 INFO:tasks.workunit.client.0.smithi129.stderr:264466: waiting osd.8 seq 193273528374 2024-03-26T00:00:33.429 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-26T00:00:33.429 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528375 -lt 193273528374 2024-03-26T00:00:34.038 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_untu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:34.038 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-26T00:00:34.038 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-493921239043 2024-03-26T00:00:34.038 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:34.038 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=493921239043 2024-03-26T00:00:34.038 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 493921239043' 2024-03-26T00:00:34.039 INFO:tasks.workunit.client.0.smithi129.stderr:264469: waiting osd.5 seq 493921239043 2024-03-26T00:00:34.039 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-26T00:00:34.039 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 493921239043 -lt 493921239043 2024-03-26T00:00:34.039 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:34.039 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855427 2024-03-26T00:00:34.039 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:34.039 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-26T00:00:34.039 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855427 2024-03-26T00:00:34.039 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:34.039 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855427 2024-03-26T00:00:34.039 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855427' 2024-03-26T00:00:34.040 INFO:tasks.workunit.client.0.smithi129.stderr:264469: waiting osd.6 seq 150323855427 2024-03-26T00:00:34.040 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-26T00:00:34.040 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855427 -lt 150323855427 2024-03-26T00:00:34.040 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:34.040 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691905 2024-03-26T00:00:34.040 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:34.040 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-26T00:00:34.040 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691905 2024-03-26T00:00:34.040 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:34.040 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691905 2024-03-26T00:00:34.040 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691905' 2024-03-26T00:00:34.040 INFO:tasks.workunit.client.0.smithi129.stderr:264469: waiting osd.7 seq 171798691905 2024-03-26T00:00:34.040 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-26T00:00:34.041 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691906 -lt 171798691905 2024-03-26T00:00:34.041 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:34.041 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528375 2024-03-26T00:00:34.041 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:34.041 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-26T00:00:34.041 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528375 2024-03-26T00:00:34.873 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stpg_stats: for s in $seqs 2024-03-26T00:00:34.874 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-446676598792 2024-03-26T00:00:34.874 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:34.874 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-26T00:00:34.874 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-446676598792 2024-03-26T00:00:34.874 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:34.874 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=446676598792 2024-03-26T00:00:34.874 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 446676598792' 2024-03-26T00:00:34.874 INFO:tasks.workunit.client.0.smithi129.stderr:264466: waiting osd.9 seq 446676598792 2024-03-26T00:00:34.874 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-26T00:00:34.875 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 446676598793 -lt 446676598792 2024-03-26T00:00:34.875 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:00:34.875 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:00:34.875 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:00:34.875 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-26T00:00:34.875 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:00:34.875 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:00:34.875 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:00:34.875 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:00:34.875 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:00:34.875 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:00:34.876 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:00:34.876 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-26T00:00:34.876 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:00:34.876 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:00:34.876 INFO:tasks.workunit.client.0.smithi129.stderr:264466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:00:34.876 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-26T00:00:34.876 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:00:34.876 INFO:tasks.workunit.client.0.smithi129.stderr:264466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:00:34.876 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-26T00:00:34.876 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-26T00:00:34.876 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 264467 2024-03-26T00:00:35.771 INFO:tasks.workunit.client.0.smithi129.stderr:ats: cut -d - -f 2 2024-03-26T00:00:35.771 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528375 2024-03-26T00:00:35.771 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528375' 2024-03-26T00:00:35.771 INFO:tasks.workunit.client.0.smithi129.stderr:264469: waiting osd.8 seq 193273528375 2024-03-26T00:00:35.771 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-26T00:00:35.771 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528375 -lt 193273528375 2024-03-26T00:00:35.771 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:35.771 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-446676598793 2024-03-26T00:00:35.772 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:35.772 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-26T00:00:35.772 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-446676598793 2024-03-26T00:00:35.772 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:35.772 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=446676598793 2024-03-26T00:00:35.772 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 446676598793' 2024-03-26T00:00:35.772 INFO:tasks.workunit.client.0.smithi129.stderr:264469: waiting osd.9 seq 446676598793 2024-03-26T00:00:35.772 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-26T00:00:35.772 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 446676598793 -lt 446676598793 2024-03-26T00:00:35.772 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:00:35.772 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:00:35.772 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:00:35.773 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-26T00:00:35.773 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:00:35.773 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:00:35.773 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:00:35.773 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:00:35.773 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:00:35.773 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:00:35.773 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:00:35.773 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-26T00:00:35.773 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:00:35.773 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:00:35.773 INFO:tasks.workunit.client.0.smithi129.stderr:264469: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:00:35.774 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-26T00:00:35.774 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:00:35.774 INFO:tasks.workunit.client.0.smithi129.stderr:264469: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:00:35.774 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-26T00:00:35.774 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-26T00:00:35.774 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-26T00:00:35.774 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-26T00:00:35.774 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-03-26T00:00:35.774 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-26T00:00:35.774 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-03-26T00:00:35.775 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-26T00:00:35.775 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-26T00:00:35.775 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-26T00:00:35.775 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-26T00:00:36.088 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-03-26T00:00:36.088 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-03-26T00:00:36.088 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-26T00:00:36.089 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-26T00:00:36.089 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-26T00:00:36.089 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:00:36.089 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:00:36.089 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-26T00:00:36.446 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-26T00:00:00.585529+0000 2024-03-26T00:00:36.447 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-26T00:00:36.734 INFO:tasks.workunit.client.0.smithi129.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-26T00:00:36.751 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-26T00:00:00.585529+0000 2024-03-26T00:00:36.751 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-26T00:00:36.751 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-26T00:00:00.585529+0000 2024-03-26T00:00:36.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-26T00:00:36.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-26T00:00:36.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:00:36.752 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-26T00:00:36.753 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-26T00:00:36.753 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:00:36.753 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:00:36.753 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-26T00:00:37.059 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:00:00.585529+0000 '>' 2024-03-26T00:00:00.585529+0000 2024-03-26T00:00:37.060 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:00:38.061 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:00:38.061 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:00:38.062 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-26T00:00:38.062 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-26T00:00:38.062 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:00:38.063 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:00:38.063 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-26T00:00:38.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:00:00.585529+0000 '>' 2024-03-26T00:00:00.585529+0000 2024-03-26T00:00:38.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:00:39.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:00:39.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:00:39.388 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-26T00:00:39.388 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-26T00:00:39.389 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:00:39.389 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:00:39.389 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-26T00:00:39.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:00:00.585529+0000 '>' 2024-03-26T00:00:00.585529+0000 2024-03-26T00:00:39.717 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:00:40.718 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:00:40.718 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:00:40.719 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-26T00:00:40.719 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-26T00:00:40.719 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:00:40.720 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:00:40.720 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-26T00:00:41.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:00:00.585529+0000 '>' 2024-03-26T00:00:00.585529+0000 2024-03-26T00:00:41.041 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:00:42.041 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:00:42.041 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:00:42.042 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-26T00:00:42.042 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-26T00:00:42.042 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:00:42.043 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:00:42.043 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-26T00:00:42.372 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:00:37.363875+0000 '>' 2024-03-26T00:00:00.585529+0000 2024-03-26T00:00:42.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-26T00:00:42.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-03-26T00:00:42.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-26T00:00:42.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-26T00:00:42.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-26T00:00:42.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 267895"' 2024-03-26T00:00:42.373 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 267895' 2024-03-26T00:00:42.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-26T00:00:42.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-26T00:00:42.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-26T00:00:42.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 267897"' 2024-03-26T00:00:42.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-26T00:00:42.374 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 267897' 2024-03-26T00:00:42.374 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-03-26T00:00:42.374 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/267898: /' 2024-03-26T00:00:42.374 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 267895 267897' 2024-03-26T00:00:42.374 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-26T00:00:42.374 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-26T00:00:42.374 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 267895 2024-03-26T00:00:42.374 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-26T00:00:42.374 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/267900: /' 2024-03-26T00:00:44.265 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:00:44.265 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:00:44.265 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-26T00:00:44.265 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:00:44.265 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-26T00:00:44.265 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:00:44.265 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:00:44.265 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-26T00:00:44.265 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:00:44.266 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-26T00:00:44.266 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:00:44.266 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:00:44.266 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:00:44.266 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:00:44.266 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:00:44.266 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:00:44.266 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-26T00:00:44.266 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:00:44.266 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:00:44.266 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-26T00:00:44.266 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:00:44.266 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-26T00:00:44.266 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-03-26T00:00:44.267 INFO:tasks.workunit.client.0.smithi129.stderr:267898: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-03-26T00:00:44.267 INFO:tasks.workunit.client.0.smithi129.stderr:267898: _ 2024-03-26T00:00:44.267 INFO:tasks.workunit.client.0.smithi129.stderr:267898: hinfo_key 2024-03-26T00:00:44.267 INFO:tasks.workunit.client.0.smithi129.stderr:267898: snapset 2024-03-26T00:00:44.267 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-26T00:00:44.267 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:00:44.267 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:00:44.267 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-26T00:00:44.267 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:00:44.267 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-26T00:00:44.267 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:00:44.269 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:00:44.270 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:00:44.270 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:00:44.270 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-26T00:00:44.270 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-26T00:00:44.270 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:00:44.270 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:00:44.270 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:00:44.270 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:00:44.270 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:00:44.270 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:00:44.270 INFO:tasks.workunit.client.0.smithi129.stderr:267898: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:00:44.270 INFO:tasks.workunit.client.0.smithi129.stderr:267898: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:00:44.270 INFO:tasks.workunit.client.0.smithi129.stderr:267898: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:00:44.270 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:00:44.271 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:00:44.271 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:00:44.271 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:00:44.271 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:00:44.271 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:00:44.271 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:00:44.271 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:00:44.271 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:00:44.271 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:00:44.271 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:00:44.271 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-26T00:00:44.271 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-26T00:00:44.271 INFO:tasks.workunit.client.0.smithi129.stderr:267898: start osd.3 2024-03-26T00:00:44.272 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:00:44.272 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:00:44.272 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-26T00:00:44.272 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:00:44.272 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-26T00:00:44.272 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:00:44.272 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:00:44.272 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-26T00:00:44.272 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:00:44.272 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-26T00:00:44.272 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:00:44.272 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:00:44.273 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:00:44.273 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:00:44.273 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:00:44.273 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:00:44.273 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-26T00:00:44.273 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:00:44.273 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:00:44.273 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-26T00:00:44.273 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:00:44.273 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-26T00:00:44.273 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-03-26T00:00:44.273 INFO:tasks.workunit.client.0.smithi129.stderr:267900: _ 2024-03-26T00:00:44.273 INFO:tasks.workunit.client.0.smithi129.stderr:267900: hinfo_key 2024-03-26T00:00:44.273 INFO:tasks.workunit.client.0.smithi129.stderr:267900: snapset 2024-03-26T00:00:44.273 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-26T00:00:44.274 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:00:44.274 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:00:44.274 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-26T00:00:44.274 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:00:44.274 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-26T00:00:44.274 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:00:44.274 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:00:44.275 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:00:44.275 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:00:44.275 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-26T00:00:44.275 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-26T00:00:44.275 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:00:44.275 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:00:44.275 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:00:44.275 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:00:44.276 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:00:44.276 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:00:44.276 INFO:tasks.workunit.client.0.smithi129.stderr:267900: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:00:44.276 INFO:tasks.workunit.client.0.smithi129.stderr:267900: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:00:44.276 INFO:tasks.workunit.client.0.smithi129.stderr:267900: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:00:44.276 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:00:44.276 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:00:44.276 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:00:44.276 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:00:44.276 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:00:44.276 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:00:44.276 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:00:44.276 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:00:44.276 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:00:44.277 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:00:44.277 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:00:44.277 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-26T00:00:44.277 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-26T00:00:44.277 INFO:tasks.workunit.client.0.smithi129.stderr:267900: start osd.5 2024-03-26T00:00:48.823 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:00:48.823 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-26T00:00:48.823 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-26T00:00:48.823 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:00:48.823 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:00:48.823 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:00:48.823 INFO:tasks.workunit.client.0.smithi129.stderr:267898: 2024-03-26T00:00:44.314+0000 7f39466117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:00:48.823 INFO:tasks.workunit.client.0.smithi129.stderr:267898: 2024-03-26T00:00:44.318+0000 7f39466117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:00:48.824 INFO:tasks.workunit.client.0.smithi129.stderr:267898: 2024-03-26T00:00:44.318+0000 7f39466117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:00:48.824 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-26T00:00:48.824 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:00:48.824 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-26T00:00:48.824 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:00:48.824 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:00:48.824 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:00:48.824 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:00:48.824 INFO:tasks.workunit.client.0.smithi129.stderr:267898: 0 2024-03-26T00:00:48.824 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:00:48.824 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-26T00:00:48.824 INFO:tasks.workunit.client.0.smithi129.stderr:267898: 2024-03-26T00:00:44.894+0000 7f39466117c0 -1 Falling back to public interface 2024-03-26T00:00:48.825 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:00:48.825 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:00:48.825 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:00:48.825 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:00:48.825 INFO:tasks.workunit.client.0.smithi129.stderr:267898: 1 2024-03-26T00:00:48.825 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:00:48.825 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-26T00:00:48.825 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:00:48.825 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:00:48.825 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:00:48.825 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:00:48.825 INFO:tasks.workunit.client.0.smithi129.stderr:267898: 2 2024-03-26T00:00:48.826 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:00:48.826 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-26T00:00:48.826 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:00:48.826 INFO:tasks.workunit.client.0.smithi129.stderr:267898: 2024-03-26T00:00:48.774+0000 7f39466117c0 -1 osd.3 117 log_to_monitors true 2024-03-26T00:00:48.826 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:00:48.826 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:00:48.826 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:00:48.826 INFO:tasks.workunit.client.0.smithi129.stderr:267898: 3 2024-03-26T00:00:48.826 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:00:49.292 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpeovery_ops 2024-03-26T00:00:49.292 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-26T00:00:49.292 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-26T00:00:49.292 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:00:49.292 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:00:49.292 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:00:49.292 INFO:tasks.workunit.client.0.smithi129.stderr:267900: 2024-03-26T00:00:44.306+0000 7f0af2f857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:00:49.292 INFO:tasks.workunit.client.0.smithi129.stderr:267900: 2024-03-26T00:00:44.310+0000 7f0af2f857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:00:49.293 INFO:tasks.workunit.client.0.smithi129.stderr:267900: 2024-03-26T00:00:44.310+0000 7f0af2f857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:00:49.293 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-26T00:00:49.293 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:00:49.293 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-26T00:00:49.293 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:00:49.293 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:00:49.293 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:00:49.293 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:00:49.293 INFO:tasks.workunit.client.0.smithi129.stderr:267900: 0 2024-03-26T00:00:49.293 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:00:49.293 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-26T00:00:49.293 INFO:tasks.workunit.client.0.smithi129.stderr:267900: 2024-03-26T00:00:44.886+0000 7f0af2f857c0 -1 Falling back to public interface 2024-03-26T00:00:49.293 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:00:49.294 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:00:49.294 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:00:49.294 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:00:49.294 INFO:tasks.workunit.client.0.smithi129.stderr:267900: 1 2024-03-26T00:00:49.294 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:00:49.294 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-26T00:00:49.294 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:00:49.294 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:00:49.294 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:00:49.294 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:00:49.294 INFO:tasks.workunit.client.0.smithi129.stderr:267900: 2 2024-03-26T00:00:49.294 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:00:49.295 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-26T00:00:49.295 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:00:49.295 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:00:49.295 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:00:49.295 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:00:49.295 INFO:tasks.workunit.client.0.smithi129.stderr:267900: 3 2024-03-26T00:00:49.295 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:00:49.295 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-26T00:00:49.295 INFO:tasks.workunit.client.0.smithi129.stderr:267900: 2024-03-26T00:00:49.042+0000 7f0af2f857c0 -1 osd.5 117 log_to_monitors true 2024-03-26T00:00:51.812 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/rs.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-26T00:00:51.812 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:00:51.812 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:00:51.812 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:00:51.813 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-26T00:00:51.813 INFO:tasks.workunit.client.0.smithi129.stderr:267898: 4 2024-03-26T00:00:51.813 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:00:51.813 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-26T00:00:51.813 INFO:tasks.workunit.client.0.smithi129.stderr:267898: osd.3 up in weight 1 up_from 122 up_thru 115 down_at 118 last_clean_interval [112,117) [v2:127.0.0.1:6842/1633258066,v1:127.0.0.1:6844/1633258066] [v2:127.0.0.1:6846/1633258066,v1:127.0.0.1:6848/1633258066] exists,up 25e4ca68-13d2-4df8-9930-c8c629bba9ef 2024-03-26T00:00:51.813 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:00:51.813 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:00:51.813 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:00:51.813 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:00:51.813 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:00:51.813 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:00:51.813 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:00:51.813 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:00:51.813 INFO:tasks.workunit.client.0.smithi129.stderr:267898: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:00:51.814 INFO:tasks.workunit.client.0.smithi129.stderr:267898: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:00:51.814 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:00:51.814 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:00:51.814 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:00:51.814 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:00:51.814 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:00:51.814 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:00:51.814 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:00:51.814 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:00:51.814 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:00:51.814 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:00:51.815 INFO:tasks.workunit.client.0.smithi129.stderr:267898: 1 2024-03-26T00:00:51.815 INFO:tasks.workunit.client.0.smithi129.stderr:267898: 2 2024-03-26T00:00:51.815 INFO:tasks.workunit.client.0.smithi129.stderr:267898: 3 2024-03-26T00:00:51.815 INFO:tasks.workunit.client.0.smithi129.stderr:267898: 4 2024-03-26T00:00:51.815 INFO:tasks.workunit.client.0.smithi129.stderr:267898: 5 2024-03-26T00:00:51.815 INFO:tasks.workunit.client.0.smithi129.stderr:267898: 6 2024-03-26T00:00:51.815 INFO:tasks.workunit.client.0.smithi129.stderr:267898: 7 2024-03-26T00:00:51.815 INFO:tasks.workunit.client.0.smithi129.stderr:267898: 8 2024-03-26T00:00:51.815 INFO:tasks.workunit.client.0.smithi129.stderr:267898: 9' 2024-03-26T00:00:51.815 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:00:51.815 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:51.815 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:00:51.815 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836605 2024-03-26T00:00:51.816 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836605 2024-03-26T00:00:51.816 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836605' 2024-03-26T00:00:51.816 INFO:tasks.workunit.client.0.smithi129.stderr:26789ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:00:51.816 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:00:51.816 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:00:51.816 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-26T00:00:51.816 INFO:tasks.workunit.client.0.smithi129.stderr:267900: 4 2024-03-26T00:00:51.816 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:00:51.816 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-26T00:00:51.816 INFO:tasks.workunit.client.0.smithi129.stderr:267900: osd.5 up in weight 1 up_from 122 up_thru 115 down_at 118 last_clean_interval [115,117) [v2:127.0.0.1:6826/2127998218,v1:127.0.0.1:6827/2127998218] [v2:127.0.0.1:6828/2127998218,v1:127.0.0.1:6829/2127998218] exists,up 67160811-ca71-46db-8586-da42b099b746 2024-03-26T00:00:51.816 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:00:51.816 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:00:51.816 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:00:51.816 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:00:51.817 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:00:51.817 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:00:51.817 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:00:51.817 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:00:51.817 INFO:tasks.workunit.client.0.smithi129.stderr:267900: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:00:51.817 INFO:tasks.workunit.client.0.smithi129.stderr:267900: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:00:51.817 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:00:51.817 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:00:51.817 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:00:51.817 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:00:51.817 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:00:51.817 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:00:51.817 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:00:51.817 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:00:51.818 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:00:51.818 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:00:51.818 INFO:tasks.workunit.client.0.smithi129.stderr:267900: 1 2024-03-26T00:00:51.818 INFO:tasks.workunit.client.0.smithi129.stderr:267900: 2 2024-03-26T00:00:51.818 INFO:tasks.workunit.client.0.smithi129.stderr:267900: 3 2024-03-26T00:00:51.818 INFO:tasks.workunit.client.0.smithi129.stderr:267900: 4 2024-03-26T00:00:51.818 INFO:tasks.workunit.client.0.smithi129.stderr:267900: 5 2024-03-26T00:00:51.818 INFO:tasks.workunit.client.0.smithi129.stderr:267900: 6 2024-03-26T00:00:51.818 INFO:tasks.workunit.client.0.smithi129.stderr:267900: 7 2024-03-26T00:00:51.818 INFO:tasks.workunit.client.0.smithi129.stderr:267900: 8 2024-03-26T00:00:51.818 INFO:tasks.workunit.client.0.smithi129.stderr:267900: 9' 2024-03-26T00:00:51.818 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:00:51.818 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:51.819 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:00:51.819 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836604 2024-03-26T00:00:51.819 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836604 2024-03-26T00:00:51.819 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836604' 2024-03-26T00:00:53.007 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd 8: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:53.007 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:00:53.007 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673073 2024-03-26T00:00:53.008 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673073 2024-03-26T00:00:53.008 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836605 1-42949673073' 2024-03-26T00:00:53.008 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:53.008 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:00:53.008 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509545 2024-03-26T00:00:53.008 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509545 2024-03-26T00:00:53.008 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836605 1-42949673073 2-64424509545' 2024-03-26T00:00:53.008 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:53.008 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-26T00:00:53.008 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=523986010115 2024-03-26T00:00:53.008 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 523986010115 2024-03-26T00:00:53.008 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836605 1-42949673073 2-64424509545 3-523986010115' 2024-03-26T00:00:53.009 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:53.009 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-26T00:00:53.009 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182491 2024-03-26T00:00:53.009 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182491 2024-03-26T00:00:53.009 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836605 1-42949673073 2-64424509545 3-523986010115 4-107374182491' 2024-03-26T00:00:53.009 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:53.009 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-26T00:00:53.009 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=523986010115 2024-03-26T00:00:53.009 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 523986010115 2024-03-26T00:00:53.009 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836605 1-42949673073 2-64424509545 3-523986010115 4-107374182491 5-523986010115' 2024-03-26T00:00:53.009 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:53.009 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-26T00:00:53.009 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855433 2024-03-26T00:00:53.010 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855433 2024-03-26T00:00:53.010 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836605 1-42949673073 2-64424509545 3-523986010115 4-107374182491 5-523986010115 6-150323855433' 2024-03-26T00:00:53.010 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:53.136 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: in $ids 2024-03-26T00:00:53.136 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:00:53.137 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673074 2024-03-26T00:00:53.137 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673074 2024-03-26T00:00:53.137 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836604 1-42949673074' 2024-03-26T00:00:53.137 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:53.137 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:00:53.137 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509544 2024-03-26T00:00:53.137 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509544 2024-03-26T00:00:53.137 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836604 1-42949673074 2-64424509544' 2024-03-26T00:00:53.137 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:53.137 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-26T00:00:53.137 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=523986010114 2024-03-26T00:00:53.137 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 523986010114 2024-03-26T00:00:53.137 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836604 1-42949673074 2-64424509544 3-523986010114' 2024-03-26T00:00:53.137 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:53.137 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-26T00:00:53.138 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182490 2024-03-26T00:00:53.138 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182490 2024-03-26T00:00:53.138 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836604 1-42949673074 2-64424509544 3-523986010114 4-107374182490' 2024-03-26T00:00:53.138 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:53.138 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-26T00:00:53.138 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=523986010114 2024-03-26T00:00:53.138 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 523986010114 2024-03-26T00:00:53.138 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836604 1-42949673074 2-64424509544 3-523986010114 4-107374182490 5-523986010114' 2024-03-26T00:00:53.138 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:53.138 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-26T00:00:53.138 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855432 2024-03-26T00:00:53.138 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855432 2024-03-26T00:00:53.138 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836604 1-42949673074 2-64424509544 3-523986010114 4-107374182490 5-523986010114 6-150323855432' 2024-03-26T00:00:53.139 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:53.139 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-26T00:00:53.707 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691910 2024-03-26T00:00:53.707 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691910 2024-03-26T00:00:53.707 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836604 1-42949673074 2-64424509544 3-523986010114 4-107374182490 5-523986010114 6-150323855432 7-171798691910' 2024-03-26T00:00:53.707 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:53.707 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-26T00:00:53.707 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528380 2024-03-26T00:00:53.707 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528380 2024-03-26T00:00:53.707 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836604 1-42949673074 2-64424509544 3-523986010114 4-107374182490 5-523986010114 6-150323855432 7-171798691910 8-193273528380' 2024-03-26T00:00:53.708 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:53.708 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-26T00:00:53.708 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=446676598798 2024-03-26T00:00:53.708 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 446676598798 2024-03-26T00:00:53.708 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836604 1-42949673074 2-64424509544 3-523986010114 4-107374182490 5-523986010114 6-150323855432 7-171798691910 8-193273528380 9-446676598798' 2024-03-26T00:00:53.708 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:53.708 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836604 2024-03-26T00:00:53.708 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:53.708 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:00:53.708 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836604 2024-03-26T00:00:53.708 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:53.708 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836604 2024-03-26T00:00:53.708 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836604' 2024-03-26T00:00:53.708 INFO:tasks.workunit.client.0.smithi129.stderr:267900: waiting osd.0 seq 21474836604 2024-03-26T00:00:53.709 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:00:53.709 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836605 -lt 21474836604 2024-03-26T00:00:53.709 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:53.709 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673074 2024-03-26T00:00:53.709 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:53.709 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:00:53.709 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673074 2024-03-26T00:00:53.709 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:53.709 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673074 2024-03-26T00:00:53.709 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673074' 2024-03-26T00:00:53.709 INFO:tasks.workunit.client.0.smithi129.stderr:267900: waiting osd.1 seq 42949673074 2024-03-26T00:00:53.715 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.shceph tell osd.7 flush_pg_stats 2024-03-26T00:00:53.715 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691911 2024-03-26T00:00:53.715 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691911 2024-03-26T00:00:53.715 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836605 1-42949673073 2-64424509545 3-523986010115 4-107374182491 5-523986010115 6-150323855433 7-171798691911' 2024-03-26T00:00:53.715 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:53.715 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-26T00:00:53.715 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528381 2024-03-26T00:00:53.715 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528381 2024-03-26T00:00:53.715 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836605 1-42949673073 2-64424509545 3-523986010115 4-107374182491 5-523986010115 6-150323855433 7-171798691911 8-193273528381' 2024-03-26T00:00:53.715 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:00:53.715 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-26T00:00:53.715 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=446676598799 2024-03-26T00:00:53.715 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 446676598799 2024-03-26T00:00:53.716 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836605 1-42949673073 2-64424509545 3-523986010115 4-107374182491 5-523986010115 6-150323855433 7-171798691911 8-193273528381 9-446676598799' 2024-03-26T00:00:53.716 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:53.716 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836605 2024-03-26T00:00:53.716 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:53.716 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:00:53.716 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836605 2024-03-26T00:00:53.716 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:53.716 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836605 2024-03-26T00:00:53.716 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836605' 2024-03-26T00:00:53.716 INFO:tasks.workunit.client.0.smithi129.stderr:267898: waiting osd.0 seq 21474836605 2024-03-26T00:00:53.716 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:00:53.716 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836605 -lt 21474836605 2024-03-26T00:00:53.716 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:53.717 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673073 2024-03-26T00:00:53.717 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:53.717 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:00:53.717 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673073 2024-03-26T00:00:53.717 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:53.717 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673073 2024-03-26T00:00:53.717 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673073' 2024-03-26T00:00:55.139 INFO:tasks.workunit.client.0.smithi129.stderr:267898: waiting osd.1 seq 42949673073 2024-03-26T00:00:55.140 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:00:55.140 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673074 -lt 42949673073 2024-03-26T00:00:55.140 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:55.140 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509545 2024-03-26T00:00:55.140 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:55.140 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:00:55.140 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509545 2024-03-26T00:00:55.140 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:55.140 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509545 2024-03-26T00:00:55.140 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509545' 2024-03-26T00:00:55.140 INFO:tasks.workunit.client.0.smithi129.stderr:267898: waiting osd.2 seq 64424509545 2024-03-26T00:00:55.140 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:00:55.140 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509545 -lt 64424509545 2024-03-26T00:00:55.140 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:55.141 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-523986010115 2024-03-26T00:00:55.141 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:55.141 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-26T00:00:55.141 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-523986010115 2024-03-26T00:00:55.141 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:55.141 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=523986010115 2024-03-26T00:00:55.141 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 523986010115' 2024-03-26T00:00:55.141 INFO:tasks.workunit.client.0.smithi129.stderr:267898: waiting osd.3 seq 523986010115 2024-03-26T00:00:55.141 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-26T00:00:55.141 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 523986010115 -lt 523986010115 2024-03-26T00:00:55.141 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:55.141 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182491 2024-03-26T00:00:55.141 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:55.141 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-26T00:00:55.141 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182491 2024-03-26T00:00:55.142 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:55.142 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182491 2024-03-26T00:00:55.142 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182491' 2024-03-26T00:00:55.142 INFO:tasks.workunit.client.0.smithi129.stderr:267898: waiting osd.4 seq 107374182491 2024-03-26T00:00:55.142 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-26T00:00:55.142 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182492 -lt 107374182491 2024-03-26T00:00:55.142 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:55.151 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cep:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:00:55.151 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673074 -lt 42949673074 2024-03-26T00:00:55.152 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:55.152 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:55.152 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509544 2024-03-26T00:00:55.152 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:00:55.152 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509544 2024-03-26T00:00:55.152 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:55.152 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509544 2024-03-26T00:00:55.152 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509544' 2024-03-26T00:00:55.152 INFO:tasks.workunit.client.0.smithi129.stderr:267900: waiting osd.2 seq 64424509544 2024-03-26T00:00:55.152 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:00:55.152 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509545 -lt 64424509544 2024-03-26T00:00:55.152 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:55.153 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-523986010114 2024-03-26T00:00:55.153 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:55.153 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-26T00:00:55.153 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-523986010114 2024-03-26T00:00:55.153 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:55.153 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=523986010114 2024-03-26T00:00:55.153 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 523986010114' 2024-03-26T00:00:55.153 INFO:tasks.workunit.client.0.smithi129.stderr:267900: waiting osd.3 seq 523986010114 2024-03-26T00:00:55.153 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-26T00:00:55.153 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 523986010115 -lt 523986010114 2024-03-26T00:00:55.154 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:55.154 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182490 2024-03-26T00:00:55.154 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:55.154 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-26T00:00:55.154 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182490 2024-03-26T00:00:55.154 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:55.154 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182490 2024-03-26T00:00:55.154 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182490' 2024-03-26T00:00:55.154 INFO:tasks.workunit.client.0.smithi129.stderr:267900: waiting osd.4 seq 107374182490 2024-03-26T00:00:55.154 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-26T00:00:55.155 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182492 -lt 107374182490 2024-03-26T00:00:55.155 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:55.155 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-523986010114 2024-03-26T00:00:56.194 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:56.194 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-26T00:00:56.194 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-523986010114 2024-03-26T00:00:56.194 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:56.194 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=523986010114 2024-03-26T00:00:56.195 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 523986010114' 2024-03-26T00:00:56.195 INFO:tasks.workunit.client.0.smithi129.stderr:267900: waiting osd.5 seq 523986010114 2024-03-26T00:00:56.195 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-26T00:00:56.195 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 523986010115 -lt 523986010114 2024-03-26T00:00:56.195 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:56.195 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:56.195 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855432 2024-03-26T00:00:56.195 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-26T00:00:56.195 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855432 2024-03-26T00:00:56.195 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:56.195 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855432 2024-03-26T00:00:56.195 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855432' 2024-03-26T00:00:56.195 INFO:tasks.workunit.client.0.smithi129.stderr:267900: waiting osd.6 seq 150323855432 2024-03-26T00:00:56.195 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-26T00:00:56.195 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855433 -lt 150323855432 2024-03-26T00:00:56.196 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:56.196 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691910 2024-03-26T00:00:56.196 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:56.196 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-26T00:00:56.196 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:56.196 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691910 2024-03-26T00:00:56.196 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691910 2024-03-26T00:00:56.196 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691910' 2024-03-26T00:00:56.196 INFO:tasks.workunit.client.0.smithi129.stderr:267900: waiting osd.7 seq 171798691910 2024-03-26T00:00:56.196 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-26T00:00:56.196 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691912 -lt 171798691910 2024-03-26T00:00:56.197 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:56.197 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528380 2024-03-26T00:00:56.197 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:56.197 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-26T00:00:56.197 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528380 2024-03-26T00:00:56.215 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: fhtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-523986010115 2024-03-26T00:00:56.215 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:56.215 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-26T00:00:56.215 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-523986010115 2024-03-26T00:00:56.215 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:56.215 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=523986010115 2024-03-26T00:00:56.215 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 523986010115' 2024-03-26T00:00:56.215 INFO:tasks.workunit.client.0.smithi129.stderr:267898: waiting osd.5 seq 523986010115 2024-03-26T00:00:56.215 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-26T00:00:56.216 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 523986010115 -lt 523986010115 2024-03-26T00:00:56.216 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:56.216 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855433 2024-03-26T00:00:56.216 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:56.216 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-26T00:00:56.216 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855433 2024-03-26T00:00:56.216 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:56.216 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855433 2024-03-26T00:00:56.216 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855433' 2024-03-26T00:00:56.216 INFO:tasks.workunit.client.0.smithi129.stderr:267898: waiting osd.6 seq 150323855433 2024-03-26T00:00:56.216 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-26T00:00:56.216 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855433 -lt 150323855433 2024-03-26T00:00:56.217 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:56.217 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691911 2024-03-26T00:00:56.217 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:56.217 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-26T00:00:56.217 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691911 2024-03-26T00:00:56.217 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:56.217 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691911 2024-03-26T00:00:56.217 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691911' 2024-03-26T00:00:56.217 INFO:tasks.workunit.client.0.smithi129.stderr:267898: waiting osd.7 seq 171798691911 2024-03-26T00:00:56.217 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-26T00:00:56.217 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691912 -lt 171798691911 2024-03-26T00:00:56.218 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:56.218 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528381 2024-03-26T00:00:56.218 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:56.218 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-26T00:00:57.958 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528381 2024-03-26T00:00:57.959 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:57.959 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528381 2024-03-26T00:00:57.959 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528381' 2024-03-26T00:00:57.959 INFO:tasks.workunit.client.0.smithi129.stderr:267898: waiting osd.8 seq 193273528381 2024-03-26T00:00:57.959 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-26T00:00:57.959 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528382 -lt 193273528381 2024-03-26T00:00:57.959 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:57.959 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-446676598799 2024-03-26T00:00:57.959 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:57.959 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-26T00:00:57.960 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-446676598799 2024-03-26T00:00:57.960 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:57.960 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=446676598799 2024-03-26T00:00:57.960 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 446676598799' 2024-03-26T00:00:57.960 INFO:tasks.workunit.client.0.smithi129.stderr:267898: waiting osd.9 seq 446676598799 2024-03-26T00:00:57.960 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-26T00:00:57.960 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 446676598800 -lt 446676598799 2024-03-26T00:00:57.960 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:00:57.960 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:00:57.960 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:00:57.960 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-26T00:00:57.960 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:00:57.961 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:00:57.961 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:00:57.961 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:00:57.961 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:00:57.961 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:00:57.961 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:00:57.961 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-26T00:00:57.961 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:00:57.961 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:00:57.961 INFO:tasks.workunit.client.0.smithi129.stderr:267898: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:00:57.961 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-26T00:00:57.961 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:00:57.962 INFO:tasks.workunit.client.0.smithi129.stderr:267898: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:00:57.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-26T00:00:57.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-26T00:00:57.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 267897 2024-03-26T00:00:57.969 INFO:tasks.workunit.client.0.smithi129.stderr:lush_pg_stats: cut -d - -f 2 2024-03-26T00:00:57.969 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528380 2024-03-26T00:00:57.969 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528380' 2024-03-26T00:00:57.969 INFO:tasks.workunit.client.0.smithi129.stderr:267900: waiting osd.8 seq 193273528380 2024-03-26T00:00:57.970 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-26T00:00:57.970 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528382 -lt 193273528380 2024-03-26T00:00:57.970 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:00:57.970 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-446676598798 2024-03-26T00:00:57.970 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:00:57.970 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-26T00:00:57.970 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-446676598798 2024-03-26T00:00:57.970 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:00:57.970 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=446676598798 2024-03-26T00:00:57.970 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 446676598798' 2024-03-26T00:00:57.970 INFO:tasks.workunit.client.0.smithi129.stderr:267900: waiting osd.9 seq 446676598798 2024-03-26T00:00:57.970 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-26T00:00:57.971 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 446676598800 -lt 446676598798 2024-03-26T00:00:57.971 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:00:57.971 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:00:57.971 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:00:57.971 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-26T00:00:57.971 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:00:57.971 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:00:57.971 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:00:57.971 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:00:57.971 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:00:57.971 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:00:57.971 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:00:57.972 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-26T00:00:57.972 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:00:57.972 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:00:57.972 INFO:tasks.workunit.client.0.smithi129.stderr:267900: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:00:57.972 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-26T00:00:57.972 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:00:57.972 INFO:tasks.workunit.client.0.smithi129.stderr:267900: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:00:57.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-26T00:00:57.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-26T00:00:57.972 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-26T00:00:57.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-26T00:00:57.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-03-26T00:00:57.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-26T00:00:57.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-26T00:00:58.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-26T00:00:58.044 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-26T00:00:58.044 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-26T00:00:58.044 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-26T00:00:58.044 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-26T00:00:58.044 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:00:58.045 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:00:58.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:00:58.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:00:58.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:00:58.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:00:58.162 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-26T00:00:58.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-26T00:00:58.162 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-26T00:00:58.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-26T00:00:58.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-26T00:00:58.162 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-26T00:00:58.163 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:00:58.163 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-26T00:00:58.163 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-26T00:00:58.163 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:00:58.164 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-26T00:00:58.164 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-26T00:00:58.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-26T00:00:58.228 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-26T00:00:58.228 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:00:58.228 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:00:58.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-26T00:00:58.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-26T00:00:58.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-26T00:00:58.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-26T00:00:58.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-26T00:00:58.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-26T00:00:58.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-26T00:00:58.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-26T00:00:58.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-26T00:00:58.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-26T00:00:58.230 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:00:58.230 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:00:58.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:00:58.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:00:58.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:00:58.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:00:58.232 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-26T00:00:58.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-26T00:00:58.233 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-26T00:00:58.233 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-26T00:00:58.233 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-26T00:00:58.234 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-26T00:00:58.235 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:00:58.235 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-26T00:00:58.235 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-26T00:00:58.235 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:00:58.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-26T00:00:58.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-26T00:00:58.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-26T00:00:58.238 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-26T00:00:58.238 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:00:58.238 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:00:58.239 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-26T00:00:58.239 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-26T00:00:58.239 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-26T00:00:58.239 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-26T00:00:58.240 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-26T00:00:58.240 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:00:58.240 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:00:58.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61636 2024-03-26T00:00:58.242 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-26T00:00:58.242 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-26T00:00:58.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-26T00:00:58.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-26T00:00:58.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_replicated td/osd-scrub-repair 2024-03-26T00:00:58.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:94: TEST_corrupt_and_repair_replicated: local dir=td/osd-scrub-repair 2024-03-26T00:00:58.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:95: TEST_corrupt_and_repair_replicated: local poolname=rbd 2024-03-26T00:00:58.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:97: TEST_corrupt_and_repair_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-03-26T00:00:58.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-26T00:00:58.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-26T00:00:58.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-26T00:00:58.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-26T00:00:58.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-26T00:00:58.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-03-26T00:00:58.723 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-26T00:00:58.723 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:00:58.724 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:00:58.724 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:00:58.724 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:00:58.725 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:00:58.725 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:00:58.725 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-03-26T00:00:58.934 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-26T00:00:58.936 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-26T00:00:58.936 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-26T00:00:58.936 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-26T00:00:58.936 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-26T00:00:58.936 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-26T00:00:58.939 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-26T00:00:58.939 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-26T00:00:58.939 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-26T00:00:58.939 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-26T00:00:58.939 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:00:58.939 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:00:58.939 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-26T00:00:58.939 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-26T00:00:58.939 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get fsid 2024-03-26T00:00:59.050 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-26T00:00:59.050 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-26T00:00:59.050 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-26T00:00:59.050 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-26T00:00:59.051 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-26T00:00:59.051 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-26T00:00:59.051 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-26T00:00:59.051 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-26T00:00:59.051 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-26T00:00:59.051 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:00:59.052 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:00:59.052 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-26T00:00:59.052 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-26T00:00:59.052 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get mon_host 2024-03-26T00:00:59.152 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:98: TEST_corrupt_and_repair_replicated: run_mgr td/osd-scrub-repair x 2024-03-26T00:00:59.152 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-26T00:00:59.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-26T00:00:59.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-26T00:00:59.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-26T00:00:59.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-26T00:00:59.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-26T00:01:00.044 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-26T00:01:00.044 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:01:00.044 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:01:00.044 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:01:00.044 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:01:00.044 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:01:00.045 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:01:00.045 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-26T00:01:00.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-26T00:01:00.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:99: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 0 2024-03-26T00:01:00.082 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-26T00:01:00.082 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-26T00:01:00.082 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-26T00:01:00.082 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-26T00:01:00.082 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:01:00.082 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:01:00.082 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:01:00.082 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:01:00.082 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:01:00.082 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-26T00:01:00.082 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-26T00:01:00.082 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-26T00:01:00.083 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-26T00:01:00.083 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:01:00.083 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-26T00:01:00.083 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:01:00.083 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:01:00.084 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:01:00.084 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:01:00.084 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:01:00.084 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:01:00.084 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:01:00.084 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:01:00.084 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-26T00:01:00.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-26T00:01:00.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:01:00.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:01:00.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:01:00.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:01:00.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:01:00.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:01:00.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-26T00:01:00.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-26T00:01:00.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-26T00:01:00.085 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-26T00:01:00.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d6614132-6c38-492d-924a-fdcb637522af 2024-03-26T00:01:00.086 INFO:tasks.workunit.client.0.smithi129.stdout:add osd0 d6614132-6c38-492d-924a-fdcb637522af 2024-03-26T00:01:00.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d6614132-6c38-492d-924a-fdcb637522af' 2024-03-26T00:01:00.086 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-26T00:01:00.104 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA8EAJmm+leBhAA5k2+pGJ28CtcSu08uf9ARg== 2024-03-26T00:01:00.104 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA8EAJmm+leBhAA5k2+pGJ28CtcSu08uf9ARg=="}' 2024-03-26T00:01:00.104 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d6614132-6c38-492d-924a-fdcb637522af -i td/osd-scrub-repair/0/new.json 2024-03-26T00:01:00.380 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-26T00:01:00.395 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-26T00:01:00.397 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA8EAJmm+leBhAA5k2+pGJ28CtcSu08uf9ARg== --osd-uuid d6614132-6c38-492d-924a-fdcb637522af 2024-03-26T00:01:00.430 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:01:00.433+0000 7f5dca5447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:01:00.447 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:01:00.449+0000 7f5dca5447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:01:00.448 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:01:00.449+0000 7f5dca5447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:01:00.448 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:01:00.449+0000 7f5dca5447c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-26T00:01:00.449 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:01:00.449+0000 7f5dca5447c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-26T00:01:00.449 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:01:00.449+0000 7f5dca5447c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-26T00:01:00.547 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:01:00.549+0000 7f5dca5447c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-26T00:01:06.928 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-26T00:01:06.928 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-26T00:01:06.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-26T00:01:06.930 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd0 key to auth repository 2024-03-26T00:01:06.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-26T00:01:07.323 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.0 2024-03-26T00:01:07.323 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-26T00:01:07.324 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:01:07.324 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-26T00:01:07.324 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-26T00:01:07.324 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-26T00:01:07.356 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:01:07.357+0000 7fd84759e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:01:07.389 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:01:07.389+0000 7fd84759e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:01:07.391 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:01:07.393+0000 7fd84759e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:01:07.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-26T00:01:07.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:01:07.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-26T00:01:07.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:01:07.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:01:07.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:01:07.665 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-26T00:01:07.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:01:07.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:01:07.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:01:07.953 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:01:07.953+0000 7fd84759e7c0 -1 Falling back to public interface 2024-03-26T00:01:08.012 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:01:09.014 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:01:09.014 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:01:09.014 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-26T00:01:09.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:01:09.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:01:09.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:01:09.335 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:01:10.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:01:10.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:01:10.337 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-26T00:01:10.338 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:01:10.338 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:01:10.338 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:01:10.594 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:01:10.597+0000 7fd84759e7c0 -1 osd.0 0 log_to_monitors true 2024-03-26T00:01:10.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:01:11.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:01:11.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:01:11.650 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-26T00:01:11.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:01:11.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:01:11.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:01:11.957 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:01:12.959 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:01:12.959 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:01:12.959 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-26T00:01:12.960 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-26T00:01:12.960 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:01:12.960 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:01:13.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:01:14.285 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:01:14.285 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:01:14.285 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-26T00:01:14.286 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-26T00:01:14.286 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:01:14.286 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:01:14.678 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:01:15.679 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:01:15.680 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:01:15.680 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-26T00:01:15.680 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-26T00:01:15.681 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:01:15.681 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:01:15.989 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:01:16.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:01:16.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:01:16.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-26T00:01:16.992 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-26T00:01:16.992 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:01:16.992 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:01:17.306 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:01:18.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:01:18.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:01:18.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-26T00:01:18.309 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-26T00:01:18.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:01:18.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:01:18.681 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:01:19.682 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:01:19.682 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:01:19.682 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-26T00:01:19.682 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-26T00:01:19.683 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:01:19.683 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:01:20.003 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:01:21.004 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:01:21.005 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:01:21.005 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-26T00:01:21.005 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-26T00:01:21.005 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:01:21.006 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:01:21.472 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:01:22.473 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:01:22.474 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:01:22.474 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-26T00:01:22.475 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-26T00:01:22.475 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:01:22.475 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:01:22.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:01:23.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:01:23.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:01:23.782 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-26T00:01:23.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-26T00:01:23.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:01:23.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:01:24.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:01:25.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:01:25.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:01:25.087 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-26T00:01:25.087 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-26T00:01:25.087 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:01:25.087 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:01:25.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:01:26.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:01:26.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:01:26.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-26T00:01:26.778 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-26T00:01:26.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:01:26.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:01:27.652 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:01:28.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:01:28.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:01:28.654 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-26T00:01:28.655 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-26T00:01:28.655 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:01:28.655 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:01:29.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:01:30.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:01:30.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:01:30.499 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-26T00:01:30.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-26T00:01:30.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:01:30.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:01:31.368 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:01:32.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:01:32.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:01:32.370 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-26T00:01:32.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-26T00:01:32.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:01:32.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:01:33.328 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:01:34.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:01:34.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:01:34.330 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-26T00:01:34.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-26T00:01:34.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:01:34.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:01:34.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:01:35.799 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:01:35.799 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:01:35.799 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-26T00:01:35.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-26T00:01:35.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:01:35.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:01:36.117 INFO:tasks.workunit.client.0.smithi129.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2270840501,v1:127.0.0.1:6803/2270840501] [v2:127.0.0.1:6804/2270840501,v1:127.0.0.1:6805/2270840501] exists,up d6614132-6c38-492d-924a-fdcb637522af 2024-03-26T00:01:36.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:01:36.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:01:36.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:01:36.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:100: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 1 2024-03-26T00:01:36.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-26T00:01:36.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-26T00:01:36.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-26T00:01:36.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-26T00:01:36.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:01:36.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:01:36.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:01:36.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:01:36.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:01:36.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:01:36.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:01:36.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-26T00:01:36.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-26T00:01:36.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:01:36.124 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-26T00:01:36.124 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:01:36.124 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:01:36.124 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:01:36.124 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:01:36.125 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:01:36.125 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:01:36.125 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:01:36.125 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:01:36.125 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-26T00:01:36.125 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-26T00:01:36.125 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:01:36.125 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:01:36.125 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:01:36.125 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:01:36.125 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:01:36.125 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:01:36.125 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-26T00:01:36.125 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-26T00:01:36.126 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:01:36.126 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-26T00:01:36.126 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=19e7cef3-b643-4a88-9c9f-8dbe95564e21 2024-03-26T00:01:36.126 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 19e7cef3-b643-4a88-9c9f-8dbe95564e21' 2024-03-26T00:01:36.126 INFO:tasks.workunit.client.0.smithi129.stdout:add osd1 19e7cef3-b643-4a88-9c9f-8dbe95564e21 2024-03-26T00:01:36.127 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-26T00:01:36.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBgEAJmRr2KCBAAYlbEk6uCXB1FNGR6ePjUAw== 2024-03-26T00:01:36.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBgEAJmRr2KCBAAYlbEk6uCXB1FNGR6ePjUAw=="}' 2024-03-26T00:01:36.141 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 19e7cef3-b643-4a88-9c9f-8dbe95564e21 -i td/osd-scrub-repair/1/new.json 2024-03-26T00:01:36.552 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-26T00:01:36.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-26T00:01:36.574 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBgEAJmRr2KCBAAYlbEk6uCXB1FNGR6ePjUAw== --osd-uuid 19e7cef3-b643-4a88-9c9f-8dbe95564e21 2024-03-26T00:01:36.606 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:01:36.608+0000 7f39c9ad47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:01:36.647 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:01:36.648+0000 7f39c9ad47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:01:36.648 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:01:36.648+0000 7f39c9ad47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:01:36.649 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:01:36.648+0000 7f39c9ad47c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-26T00:01:36.649 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:01:36.648+0000 7f39c9ad47c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-26T00:01:36.649 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:01:36.648+0000 7f39c9ad47c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-26T00:01:36.717 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:01:36.716+0000 7f39c9ad47c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-26T00:01:43.440 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-26T00:01:43.440 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-26T00:01:43.441 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd1 key to auth repository 2024-03-26T00:01:43.441 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-26T00:01:43.441 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-26T00:01:43.821 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.1 2024-03-26T00:01:43.821 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-26T00:01:43.822 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:01:43.822 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-26T00:01:43.822 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-26T00:01:43.822 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-26T00:01:43.853 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:01:43.852+0000 7fce371fd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:01:43.855 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:01:43.856+0000 7fce371fd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:01:43.856 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:01:43.856+0000 7fce371fd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:01:44.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-26T00:01:44.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:01:44.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:01:44.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:01:44.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:01:44.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:01:44.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:01:44.173 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-26T00:01:44.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:01:44.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:01:44.416 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:01:44.416+0000 7fce371fd7c0 -1 Falling back to public interface 2024-03-26T00:01:44.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:01:45.529 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:01:45.530 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:01:45.530 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-26T00:01:45.531 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:01:45.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:01:45.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:01:45.852 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:01:46.854 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:01:46.854 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:01:46.855 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-26T00:01:46.855 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:01:46.855 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:01:46.855 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:01:47.103 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:01:47.104+0000 7fce371fd7c0 -1 osd.1 0 log_to_monitors true 2024-03-26T00:01:47.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:01:48.190 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:01:48.190 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:01:48.191 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-26T00:01:48.191 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:01:48.191 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:01:48.191 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:01:48.558 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:01:49.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:01:49.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:01:49.560 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-26T00:01:49.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-26T00:01:49.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:01:49.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:01:49.945 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:01:50.947 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:01:50.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:01:50.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-26T00:01:50.948 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-26T00:01:50.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:01:50.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:01:51.366 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:01:52.367 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:01:52.367 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:01:52.368 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-26T00:01:52.368 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-26T00:01:52.368 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:01:52.368 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:01:52.709 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:01:53.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:01:54.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:01:54.113 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-26T00:01:54.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-26T00:01:54.114 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:01:54.114 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:01:54.114 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:01:55.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:01:55.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:01:55.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-26T00:01:55.034 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-26T00:01:55.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:01:55.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:01:55.410 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:01:56.412 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:01:56.412 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:01:56.412 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-26T00:01:56.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-26T00:01:56.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:01:56.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:01:56.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:01:57.736 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:01:57.737 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:01:57.737 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-26T00:01:57.738 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-26T00:01:57.738 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:01:57.739 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:01:58.103 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:01:59.104 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:01:59.105 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:01:59.105 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-26T00:01:59.105 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-26T00:01:59.105 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:01:59.105 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:01:59.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:02:00.552 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:02:00.552 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:02:00.552 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-26T00:02:00.553 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-26T00:02:00.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:02:00.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:02:01.016 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:02:02.018 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:02:02.018 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:02:02.019 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-26T00:02:02.019 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-26T00:02:02.019 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:02:02.019 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:02:02.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:02:03.632 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:02:03.632 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:02:03.633 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-26T00:02:03.633 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-26T00:02:03.633 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:02:03.633 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:02:04.322 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:02:05.324 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:02:05.324 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:02:05.324 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-26T00:02:05.325 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-26T00:02:05.325 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:02:05.325 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:02:06.196 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:02:07.198 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:02:07.198 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:02:07.198 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-26T00:02:07.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-26T00:02:07.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:02:07.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:02:07.542 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:02:08.544 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:02:08.544 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:02:08.544 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-26T00:02:08.545 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-26T00:02:08.545 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:02:08.545 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:02:09.052 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:02:10.054 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:02:10.054 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:02:10.055 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-26T00:02:10.055 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-26T00:02:10.055 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:02:10.055 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:02:10.682 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:02:11.683 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:02:11.684 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:02:11.684 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-26T00:02:11.684 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-26T00:02:11.684 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:02:11.685 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:02:12.396 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:02:13.397 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:02:13.397 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:02:13.398 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-26T00:02:13.398 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-26T00:02:13.398 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:02:13.398 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:02:13.773 INFO:tasks.workunit.client.0.smithi129.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/676198489,v1:127.0.0.1:6811/676198489] [v2:127.0.0.1:6812/676198489,v1:127.0.0.1:6813/676198489] exists,up 19e7cef3-b643-4a88-9c9f-8dbe95564e21 2024-03-26T00:02:13.773 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:02:13.774 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:02:13.774 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:02:13.774 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:101: TEST_corrupt_and_repair_replicated: create_rbd_pool 2024-03-26T00:02:13.774 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-26T00:02:14.148 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'rbd' does not exist 2024-03-26T00:02:14.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-26T00:02:14.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-26T00:02:14.722 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'rbd' created 2024-03-26T00:02:14.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-26T00:02:15.748 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-26T00:02:17.819 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:102: TEST_corrupt_and_repair_replicated: wait_for_clean 2024-03-26T00:02:17.819 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:02:17.819 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:02:17.819 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:02:17.819 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:02:17.820 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:02:17.820 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:02:17.821 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:02:17.821 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:02:17.821 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:02:17.945 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:02:17.945 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:02:17.945 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:02:17.945 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:02:17.945 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:02:17.945 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:02:18.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:02:18.257 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:02:18.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:02:18.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:02:18.258 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:02:18.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836500 2024-03-26T00:02:18.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836500 2024-03-26T00:02:18.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500' 2024-03-26T00:02:18.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:02:18.434 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:02:18.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672971 2024-03-26T00:02:18.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672971 2024-03-26T00:02:18.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672971' 2024-03-26T00:02:18.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:02:18.579 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836500 2024-03-26T00:02:18.579 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:02:18.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:02:18.580 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836500 2024-03-26T00:02:18.580 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:02:18.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836500 2024-03-26T00:02:18.581 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 21474836500 2024-03-26T00:02:18.582 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-03-26T00:02:18.582 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:02:18.880 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836500 2024-03-26T00:02:18.881 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:02:18.881 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:02:18.882 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672971 2024-03-26T00:02:18.883 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:02:18.883 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672971 2024-03-26T00:02:18.884 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:02:18.886 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672971 2024-03-26T00:02:18.887 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 42949672971 2024-03-26T00:02:18.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2024-03-26T00:02:18.887 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:02:19.210 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672971 -lt 42949672971 2024-03-26T00:02:19.214 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:02:19.214 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:02:19.214 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:02:19.582 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-26T00:02:19.582 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:02:19.583 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:02:19.583 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:02:19.583 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:02:19.583 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:02:19.583 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:02:19.583 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:02:19.876 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-26T00:02:19.876 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:02:19.876 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:02:19.877 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:02:20.245 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-26T00:02:20.245 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:02:20.245 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:02:20.245 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:104: TEST_corrupt_and_repair_replicated: add_something td/osd-scrub-repair rbd 2024-03-26T00:02:20.245 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-26T00:02:20.245 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2024-03-26T00:02:20.245 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-03-26T00:02:20.245 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-26T00:02:20.245 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-26T00:02:20.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-26T00:02:20.650 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-26T00:02:20.672 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-26T00:02:21.077 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-26T00:02:21.096 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-26T00:02:21.096 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-26T00:02:21.096 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-26T00:02:21.393 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:105: TEST_corrupt_and_repair_replicated: get_not_primary rbd SOMETHING 2024-03-26T00:02:21.394 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=rbd 2024-03-26T00:02:21.394 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-26T00:02:21.394 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary rbd SOMETHING 2024-03-26T00:02:21.394 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-03-26T00:02:21.394 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-26T00:02:21.395 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-03-26T00:02:21.395 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-26T00:02:21.727 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-26T00:02:21.728 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map rbd SOMETHING 2024-03-26T00:02:21.728 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-26T00:02:22.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:105: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2024-03-26T00:02:22.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-26T00:02:22.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-03-26T00:02:22.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=0 2024-03-26T00:02:22.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-03-26T00:02:22.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:02:22.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:02:22.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-26T00:02:22.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:02:22.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-03-26T00:02:22.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:02:22.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:02:22.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-26T00:02:22.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:02:22.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-26T00:02:22.046 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:02:22.046 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:02:22.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:02:22.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:02:22.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:02:22.354 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:02:22.355 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-03-26T00:02:22.355 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:02:22.355 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:02:22.355 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:02:22.355 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:02:22.355 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:02:22.355 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-03-26T00:02:24.023 INFO:tasks.workunit.client.0.smithi129.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-03-26T00:02:24.557 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-26T00:02:24.557 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:02:24.557 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:02:24.558 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-26T00:02:24.558 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:02:24.558 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:02:24.558 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:02:24.558 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:02:24.558 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:02:24.559 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:02:24.559 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-26T00:02:24.559 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-26T00:02:24.559 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:02:24.559 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:02:24.559 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:02:24.560 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:02:24.560 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:02:24.560 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:02:24.564 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:02:24.564 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:02:24.564 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:02:24.564 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:02:24.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:02:24.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:02:24.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:02:24.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:02:24.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:02:24.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:02:24.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:02:24.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:02:24.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:02:24.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:02:24.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-26T00:02:24.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-26T00:02:24.565 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.0 2024-03-26T00:02:24.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:02:24.566 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-26T00:02:24.566 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-26T00:02:24.566 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:02:24.566 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:02:24.566 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:02:24.595 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:02:24.595+0000 7ff1e07497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:02:24.598 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:02:24.595+0000 7ff1e07497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:02:24.599 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:02:24.599+0000 7ff1e07497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:02:24.886 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-26T00:02:24.886 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:02:24.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-26T00:02:24.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:02:24.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:02:24.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:02:24.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:02:24.887 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:02:24.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:02:24.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:02:25.132 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:02:25.131+0000 7ff1e07497c0 -1 Falling back to public interface 2024-03-26T00:02:25.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:02:26.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:02:26.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:02:26.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:02:26.202 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:02:26.202 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:02:26.202 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:02:26.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:02:27.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:02:27.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:02:27.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:02:27.507 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:02:27.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:02:27.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:02:27.821 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:02:27.830 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:02:27.831+0000 7ff1e07497c0 -1 osd.0 21 log_to_monitors true 2024-03-26T00:02:28.823 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:02:28.823 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:02:28.824 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:02:28.824 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:02:28.824 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:02:28.824 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:02:29.342 INFO:tasks.workunit.client.0.smithi129.stderr:osd.0 up in weight 1 up_from 25 up_thru 25 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/1631051155,v1:127.0.0.1:6803/1631051155] [v2:127.0.0.1:6804/1631051155,v1:127.0.0.1:6805/1631051155] exists,up d6614132-6c38-492d-924a-fdcb637522af 2024-03-26T00:02:29.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:02:29.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:02:29.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:02:29.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:02:29.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:02:29.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:02:29.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:02:29.344 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:02:29.345 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:02:29.345 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:02:29.345 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:02:29.345 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:02:29.346 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:02:29.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:02:29.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:02:29.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:02:29.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:02:29.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:02:29.459 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:02:29.773 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:02:29.773 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:02:29.773 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:02:29.773 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:02:29.774 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:02:29.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-03-26T00:02:29.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-03-26T00:02:29.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-03-26T00:02:29.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:02:29.949 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:02:30.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672975 2024-03-26T00:02:30.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672975 2024-03-26T00:02:30.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672975' 2024-03-26T00:02:30.143 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:02:30.143 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-03-26T00:02:30.143 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:02:30.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:02:30.146 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-03-26T00:02:30.146 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:02:30.148 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-03-26T00:02:30.148 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-03-26T00:02:30.148 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 107374182402 2024-03-26T00:02:30.149 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:02:30.468 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-03-26T00:02:30.468 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:02:31.470 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:02:31.471 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:02:31.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-03-26T00:02:31.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:02:31.785 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672975 2024-03-26T00:02:31.785 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:02:31.787 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:02:31.788 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672975 2024-03-26T00:02:31.788 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:02:31.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672975 2024-03-26T00:02:31.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-03-26T00:02:31.789 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 42949672975 2024-03-26T00:02:31.789 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:02:32.131 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672975 -lt 42949672975 2024-03-26T00:02:32.132 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:02:32.132 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:02:32.132 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:02:32.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-26T00:02:32.542 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:02:32.543 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:02:32.543 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:02:32.543 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:02:32.543 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:02:32.544 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:02:32.544 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:02:32.913 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-26T00:02:32.913 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:02:32.914 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:02:32.914 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:02:33.286 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-26T00:02:33.286 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:02:33.286 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:02:33.286 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-03-26T00:02:33.286 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-03-26T00:02:33.286 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-26T00:02:33.287 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-03-26T00:02:33.287 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-26T00:02:33.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-03-26T00:02:33.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-03-26T00:02:33.600 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.3 2024-03-26T00:02:33.600 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.3 2024-03-26T00:02:33.600 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-26T00:02:33.600 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:02:33.600 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:02:33.600 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-26T00:02:33.933 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-26T00:02:14.582845+0000 2024-03-26T00:02:33.933 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.3 2024-03-26T00:02:34.220 INFO:tasks.workunit.client.0.smithi129.stderr:instructing pg 1.3 on osd.1 to repair 2024-03-26T00:02:34.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.3 2024-03-26T00:02:14.582845+0000 2024-03-26T00:02:34.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.3 2024-03-26T00:02:34.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-26T00:02:14.582845+0000 2024-03-26T00:02:34.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-26T00:02:34.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-26T00:02:34.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:02:34.238 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-26T00:02:34.238 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-26T00:02:34.238 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:02:34.238 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:02:34.238 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-26T00:02:34.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:02:14.582845+0000 '>' 2024-03-26T00:02:14.582845+0000 2024-03-26T00:02:34.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:02:35.529 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:02:35.529 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:02:35.530 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-26T00:02:35.530 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-26T00:02:35.530 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:02:35.531 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-26T00:02:35.531 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:02:35.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:02:14.582845+0000 '>' 2024-03-26T00:02:14.582845+0000 2024-03-26T00:02:35.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:02:36.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:02:36.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:02:36.834 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-26T00:02:36.834 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-26T00:02:36.834 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:02:36.834 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:02:36.835 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-26T00:02:37.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:02:14.582845+0000 '>' 2024-03-26T00:02:14.582845+0000 2024-03-26T00:02:37.141 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:02:38.141 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:02:38.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:02:38.142 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-26T00:02:38.142 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-26T00:02:38.142 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:02:38.143 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:02:38.143 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-26T00:02:38.474 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:02:14.582845+0000 '>' 2024-03-26T00:02:14.582845+0000 2024-03-26T00:02:38.474 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:02:39.475 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:02:39.475 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:02:39.476 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-26T00:02:39.476 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-26T00:02:39.476 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:02:39.477 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:02:39.477 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-26T00:02:39.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:02:14.582845+0000 '>' 2024-03-26T00:02:14.582845+0000 2024-03-26T00:02:39.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:02:40.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:02:40.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:02:40.780 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-26T00:02:40.781 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-26T00:02:40.781 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:02:40.781 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:02:40.781 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-26T00:02:41.099 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:02:35.206836+0000 '>' 2024-03-26T00:02:14.582845+0000 2024-03-26T00:02:41.099 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-26T00:02:41.099 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-26T00:02:41.099 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:02:41.099 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:02:41.099 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-26T00:02:41.099 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:02:41.100 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-26T00:02:41.100 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:02:41.100 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:02:41.100 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-26T00:02:41.100 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:02:41.100 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-26T00:02:41.100 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:02:41.100 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:02:41.100 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:02:41.100 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:02:41.100 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:02:41.408 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:02:41.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-26T00:02:41.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:02:41.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:02:41.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:02:41.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:02:41.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:02:41.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-26T00:02:41.733 INFO:tasks.workunit.client.0.smithi129.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-03-26T00:02:41.734 INFO:tasks.workunit.client.0.smithi129.stdout:_ 2024-03-26T00:02:41.734 INFO:tasks.workunit.client.0.smithi129.stdout:snapset 2024-03-26T00:02:42.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-26T00:02:42.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:02:42.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:02:42.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-26T00:02:42.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:02:42.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:02:42.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:02:42.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:02:42.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:02:42.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:02:42.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:02:42.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:02:42.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:02:42.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:02:42.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:02:42.034 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:02:42.034 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:02:42.034 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:02:42.034 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:02:42.034 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:02:42.034 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:02:42.036 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:02:42.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:02:42.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:02:42.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:02:42.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:02:42.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:02:42.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:02:42.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:02:42.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:02:42.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:02:42.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:02:42.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:02:42.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-26T00:02:42.037 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.1 2024-03-26T00:02:42.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:02:42.037 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-26T00:02:42.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-26T00:02:42.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:02:42.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:02:42.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:02:42.069 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:02:42.067+0000 7f22d6b827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:02:42.072 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:02:42.071+0000 7f22d6b827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:02:42.074 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:02:42.071+0000 7f22d6b827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:02:42.358 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-26T00:02:42.358 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:02:42.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:02:42.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:02:42.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:02:42.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:02:42.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:02:42.359 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:02:42.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:02:42.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:02:42.605 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:02:42.603+0000 7f22d6b827c0 -1 Falling back to public interface 2024-03-26T00:02:42.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:02:43.700 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:02:43.700 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:02:43.700 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:02:43.700 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:02:43.700 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:02:43.700 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:02:44.005 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:02:45.007 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:02:45.007 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:02:45.007 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:02:45.007 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:02:45.008 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:02:45.008 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:02:45.369 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:02:45.803 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:02:45.803+0000 7f22d6b827c0 -1 osd.1 26 log_to_monitors true 2024-03-26T00:02:46.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:02:46.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:02:46.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:02:46.371 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:02:46.372 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:02:46.372 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:02:46.778 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1 up in weight 1 up_from 30 up_thru 30 down_at 27 last_clean_interval [10,26) [v2:127.0.0.1:6810/1608236139,v1:127.0.0.1:6811/1608236139] [v2:127.0.0.1:6812/1608236139,v1:127.0.0.1:6813/1608236139] exists,up 19e7cef3-b643-4a88-9c9f-8dbe95564e21 2024-03-26T00:02:46.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:02:46.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:02:46.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:02:46.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:02:46.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:02:46.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:02:46.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:02:46.779 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:02:46.780 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:02:46.780 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:02:46.780 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:02:46.780 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:02:46.780 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:02:46.890 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:02:46.890 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:02:46.890 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:02:46.891 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:02:46.891 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:02:46.891 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:02:47.234 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:02:47.234 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:02:47.234 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:02:47.234 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:02:47.235 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:02:47.402 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182406 2024-03-26T00:02:47.402 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182406 2024-03-26T00:02:47.402 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182406' 2024-03-26T00:02:47.402 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:02:47.402 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:02:47.544 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-03-26T00:02:47.544 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-03-26T00:02:47.544 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182406 1-128849018882' 2024-03-26T00:02:47.544 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:02:47.545 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182406 2024-03-26T00:02:47.545 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:02:47.545 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:02:47.546 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182406 2024-03-26T00:02:47.546 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:02:47.547 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182406 2024-03-26T00:02:47.547 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182406' 2024-03-26T00:02:47.547 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 107374182406 2024-03-26T00:02:47.548 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:02:47.841 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182405 -lt 107374182406 2024-03-26T00:02:47.841 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:02:48.842 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:02:48.843 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:02:49.193 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182407 -lt 107374182406 2024-03-26T00:02:49.193 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:02:49.194 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-128849018882 2024-03-26T00:02:49.194 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:02:49.195 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:02:49.196 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-128849018882 2024-03-26T00:02:49.197 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:02:49.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-03-26T00:02:49.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 128849018882' 2024-03-26T00:02:49.199 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 128849018882 2024-03-26T00:02:49.199 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:02:49.548 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-03-26T00:02:49.549 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:02:49.549 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:02:49.549 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:02:49.968 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-26T00:02:49.968 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:02:49.969 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:02:49.969 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:02:49.969 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:02:49.969 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:02:49.969 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:02:49.969 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:02:50.311 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-26T00:02:50.311 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:02:50.312 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:02:50.312 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:02:50.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-26T00:02:50.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:02:50.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:02:50.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-03-26T00:02:50.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-26T00:02:50.760 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:107: TEST_corrupt_and_repair_replicated: get_primary rbd SOMETHING 2024-03-26T00:02:50.760 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-03-26T00:02:50.760 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-26T00:02:50.760 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-03-26T00:02:50.761 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-26T00:02:51.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:107: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 1 2024-03-26T00:02:51.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-26T00:02:51.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-03-26T00:02:51.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-03-26T00:02:51.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-26T00:02:51.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:02:51.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:02:51.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-26T00:02:51.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:02:51.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-26T00:02:51.079 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:02:51.079 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:02:51.079 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-26T00:02:51.079 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:02:51.079 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-26T00:02:51.079 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:02:51.079 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:02:51.080 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:02:51.080 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:02:51.080 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:02:51.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:02:51.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-26T00:02:51.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:02:51.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:02:51.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:02:51.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:02:51.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:02:51.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-26T00:02:52.972 INFO:tasks.workunit.client.0.smithi129.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-03-26T00:02:53.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-26T00:02:53.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:02:53.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:02:53.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-26T00:02:53.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:02:53.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:02:53.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:02:53.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:02:53.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:02:53.514 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:02:53.514 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:02:53.514 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:02:53.514 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:02:53.514 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:02:53.514 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:02:53.514 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:02:53.514 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:02:53.514 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:02:53.514 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:02:53.514 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:02:53.514 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:02:53.516 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:02:53.516 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:02:53.516 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:02:53.516 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:02:53.516 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:02:53.516 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:02:53.516 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:02:53.516 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:02:53.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:02:53.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:02:53.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:02:53.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:02:53.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-26T00:02:53.517 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.1 2024-03-26T00:02:53.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:02:53.517 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-26T00:02:53.518 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-26T00:02:53.518 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:02:53.518 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:02:53.518 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:02:53.549 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:02:53.547+0000 7f6090db17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:02:53.552 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:02:53.551+0000 7f6090db17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:02:53.554 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:02:53.551+0000 7f6090db17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:02:53.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-26T00:02:53.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:02:53.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:02:53.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:02:53.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:02:53.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:02:53.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:02:53.831 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:02:53.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:02:53.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:02:54.101 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:02:54.099+0000 7f6090db17c0 -1 Falling back to public interface 2024-03-26T00:02:54.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:02:55.159 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:02:55.159 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:02:55.159 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:02:55.159 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:02:55.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:02:55.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:02:55.482 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:02:56.483 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:02:56.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:02:56.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:02:56.484 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:02:56.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:02:56.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:02:56.687 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:02:56.687+0000 7f6090db17c0 -1 osd.1 31 log_to_monitors true 2024-03-26T00:02:56.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:02:57.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:02:57.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:02:57.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:02:57.832 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:02:57.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:02:57.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:02:58.178 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1 up in weight 1 up_from 35 up_thru 35 down_at 32 last_clean_interval [30,31) [v2:127.0.0.1:6810/1178920011,v1:127.0.0.1:6811/1178920011] [v2:127.0.0.1:6812/1178920011,v1:127.0.0.1:6813/1178920011] exists,up 19e7cef3-b643-4a88-9c9f-8dbe95564e21 2024-03-26T00:02:58.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:02:58.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:02:58.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:02:58.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:02:58.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:02:58.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:02:58.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:02:58.179 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:02:58.179 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:02:58.179 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:02:58.179 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:02:58.180 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:02:58.180 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:02:58.295 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:02:58.295 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:02:58.295 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:02:58.295 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:02:58.296 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:02:58.296 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:02:58.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:02:58.615 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:02:58.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:02:58.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:02:58.615 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:02:58.856 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182410 2024-03-26T00:02:58.856 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182410 2024-03-26T00:02:58.856 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182410' 2024-03-26T00:02:58.856 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:02:58.856 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:02:59.008 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-03-26T00:02:59.008 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-03-26T00:02:59.008 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182410 1-150323855362' 2024-03-26T00:02:59.008 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:02:59.008 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182410 2024-03-26T00:02:59.009 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:02:59.009 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:02:59.010 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182410 2024-03-26T00:02:59.010 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:02:59.011 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182410 2024-03-26T00:02:59.011 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182410' 2024-03-26T00:02:59.011 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 107374182410 2024-03-26T00:02:59.011 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:02:59.308 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182409 -lt 107374182410 2024-03-26T00:02:59.308 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:03:00.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:03:00.311 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:03:00.622 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182409 -lt 107374182410 2024-03-26T00:03:00.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:03:01.624 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-26T00:03:01.625 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:03:01.946 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182410 -lt 107374182410 2024-03-26T00:03:01.946 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:03:01.947 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855362 2024-03-26T00:03:01.947 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:03:01.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:03:01.949 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:03:01.949 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855362 2024-03-26T00:03:01.950 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-03-26T00:03:01.950 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855362' 2024-03-26T00:03:01.951 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 150323855362 2024-03-26T00:03:01.951 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:03:02.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-03-26T00:03:02.266 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:03:02.266 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:03:02.267 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:03:02.700 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-26T00:03:02.700 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:03:02.701 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:03:02.701 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:03:02.701 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:03:02.701 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:03:02.701 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:03:02.701 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:03:03.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-26T00:03:03.011 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:03:03.011 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:03:03.011 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:03:03.375 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-26T00:03:03.375 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:03:03.375 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:03:03.376 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-03-26T00:03:03.376 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-03-26T00:03:03.376 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-26T00:03:03.376 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-03-26T00:03:03.376 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-26T00:03:03.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-03-26T00:03:03.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-03-26T00:03:03.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.3 2024-03-26T00:03:03.706 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.3 2024-03-26T00:03:03.706 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-26T00:03:03.706 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:03:03.707 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:03:03.707 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-26T00:03:04.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-26T00:02:35.206836+0000 2024-03-26T00:03:04.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.3 2024-03-26T00:03:04.315 INFO:tasks.workunit.client.0.smithi129.stderr:instructing pg 1.3 on osd.1 to repair 2024-03-26T00:03:04.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.3 2024-03-26T00:02:35.206836+0000 2024-03-26T00:03:04.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.3 2024-03-26T00:03:04.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-26T00:02:35.206836+0000 2024-03-26T00:03:04.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-26T00:03:04.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-26T00:03:04.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:03:04.331 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-26T00:03:04.332 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-26T00:03:04.332 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:03:04.332 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:03:04.332 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-26T00:03:04.642 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:02:35.206836+0000 '>' 2024-03-26T00:02:35.206836+0000 2024-03-26T00:03:04.642 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:03:05.643 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:03:05.644 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:03:05.644 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-26T00:03:05.644 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-26T00:03:05.644 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:03:05.645 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:03:05.645 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-26T00:03:05.968 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:02:35.206836+0000 '>' 2024-03-26T00:02:35.206836+0000 2024-03-26T00:03:05.969 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:03:06.970 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:03:06.970 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:03:06.971 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-26T00:03:06.971 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-26T00:03:06.971 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:03:06.971 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:03:06.972 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-26T00:03:07.282 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:02:35.206836+0000 '>' 2024-03-26T00:02:35.206836+0000 2024-03-26T00:03:07.282 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:03:08.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:03:08.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:03:08.284 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-26T00:03:08.284 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-26T00:03:08.285 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:03:08.285 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:03:08.285 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-26T00:03:08.602 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:02:35.206836+0000 '>' 2024-03-26T00:02:35.206836+0000 2024-03-26T00:03:08.603 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:03:09.604 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:03:09.604 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:03:09.605 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-26T00:03:09.605 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-26T00:03:09.605 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:03:09.605 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:03:09.606 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-26T00:03:09.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:03:04.787169+0000 '>' 2024-03-26T00:02:35.206836+0000 2024-03-26T00:03:09.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-26T00:03:09.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-26T00:03:09.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:03:09.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:03:09.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-26T00:03:09.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:03:09.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-26T00:03:09.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:03:09.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:03:09.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-26T00:03:09.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:03:09.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-26T00:03:09.920 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:03:09.921 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:03:09.921 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:03:09.921 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:03:09.921 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:03:10.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:03:10.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-26T00:03:10.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:03:10.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:03:10.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:03:10.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:03:10.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:03:10.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-26T00:03:10.484 INFO:tasks.workunit.client.0.smithi129.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-03-26T00:03:10.485 INFO:tasks.workunit.client.0.smithi129.stdout:_ 2024-03-26T00:03:10.485 INFO:tasks.workunit.client.0.smithi129.stdout:snapset 2024-03-26T00:03:10.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-26T00:03:10.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:03:10.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:03:10.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-26T00:03:10.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:03:10.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:03:10.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:03:10.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:03:10.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:03:10.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:03:10.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:03:10.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:03:10.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:03:10.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:03:10.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:03:10.781 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:03:10.781 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:03:10.781 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:03:10.781 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:03:10.781 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:03:10.781 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:03:10.781 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:03:10.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:03:10.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:03:10.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:03:10.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:03:10.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:03:10.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:03:10.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:03:10.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:03:10.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:03:10.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:03:10.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:03:10.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-26T00:03:10.783 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.1 2024-03-26T00:03:10.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:03:10.783 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-26T00:03:10.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-26T00:03:10.785 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:03:10.785 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:03:10.786 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:03:10.816 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:03:10.814+0000 7fdd59bb57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:03:10.819 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:03:10.818+0000 7fdd59bb57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:03:10.820 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:03:10.818+0000 7fdd59bb57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:03:11.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-26T00:03:11.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:03:11.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:03:11.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:03:11.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:03:11.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:03:11.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:03:11.103 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:03:11.103 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:03:11.103 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:03:11.360 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:03:11.358+0000 7fdd59bb57c0 -1 Falling back to public interface 2024-03-26T00:03:11.441 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:03:12.443 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:03:12.443 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:03:12.443 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:03:12.443 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:03:12.444 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:03:12.444 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:03:12.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:03:13.747 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:03:13.748 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:03:13.748 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:03:13.748 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:03:13.748 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:03:13.748 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:03:14.064 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:03:14.409 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:03:14.406+0000 7fdd59bb57c0 -1 osd.1 37 log_to_monitors true 2024-03-26T00:03:15.066 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:03:15.066 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:03:15.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:03:15.067 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:03:15.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:03:15.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:03:15.577 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1 up in weight 1 up_from 41 up_thru 41 down_at 38 last_clean_interval [35,37) [v2:127.0.0.1:6810/284142034,v1:127.0.0.1:6811/284142034] [v2:127.0.0.1:6812/284142034,v1:127.0.0.1:6813/284142034] exists,up 19e7cef3-b643-4a88-9c9f-8dbe95564e21 2024-03-26T00:03:15.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:03:15.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:03:15.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:03:15.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:03:15.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:03:15.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:03:15.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:03:15.579 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:03:15.579 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:03:15.579 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:03:15.579 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:03:15.579 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:03:15.579 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:03:15.692 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:03:15.692 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:03:15.692 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:03:15.692 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:03:15.692 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:03:15.693 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:03:16.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:03:16.010 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:03:16.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:03:16.011 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:03:16.011 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:03:16.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182414 2024-03-26T00:03:16.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182414 2024-03-26T00:03:16.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182414' 2024-03-26T00:03:16.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:03:16.258 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:03:16.405 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659138 2024-03-26T00:03:16.405 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659138 2024-03-26T00:03:16.405 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182414 1-176093659138' 2024-03-26T00:03:16.405 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:03:16.406 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182414 2024-03-26T00:03:16.406 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:03:16.407 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:03:16.408 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182414 2024-03-26T00:03:16.408 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:03:16.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182414 2024-03-26T00:03:16.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182414' 2024-03-26T00:03:16.409 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 107374182414 2024-03-26T00:03:16.409 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:03:16.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182413 -lt 107374182414 2024-03-26T00:03:16.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:03:17.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:03:17.712 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:03:18.025 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182414 -lt 107374182414 2024-03-26T00:03:18.025 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:03:18.026 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-176093659138 2024-03-26T00:03:18.026 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:03:18.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:03:18.028 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-176093659138 2024-03-26T00:03:18.028 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:03:18.030 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659138 2024-03-26T00:03:18.030 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 176093659138' 2024-03-26T00:03:18.030 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 176093659138 2024-03-26T00:03:18.030 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:03:18.339 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659138 -lt 176093659138 2024-03-26T00:03:18.341 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:03:18.341 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:03:18.341 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:03:18.736 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-26T00:03:18.737 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:03:18.737 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:03:18.737 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:03:18.737 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:03:18.738 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:03:18.738 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:03:18.738 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:03:19.041 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-26T00:03:19.042 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:03:19.042 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:03:19.043 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:03:19.422 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-26T00:03:19.422 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:03:19.422 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:03:19.423 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-03-26T00:03:19.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-26T00:03:19.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-26T00:03:19.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-26T00:03:19.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-26T00:03:19.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-26T00:03:19.460 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:03:19.460 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:03:19.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:03:19.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:03:19.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:03:19.770 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:03:19.770 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-26T00:03:19.771 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-26T00:03:19.772 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-26T00:03:19.773 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-26T00:03:19.773 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-26T00:03:19.773 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-26T00:03:19.775 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:03:19.775 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-26T00:03:19.775 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-26T00:03:19.776 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:03:19.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-26T00:03:19.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-26T00:03:19.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-26T00:03:19.797 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-26T00:03:19.797 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:03:19.797 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:03:19.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-26T00:03:19.799 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-26T00:03:19.799 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-26T00:03:19.799 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-26T00:03:19.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-26T00:03:19.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-26T00:03:19.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-26T00:03:19.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-26T00:03:19.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-26T00:03:19.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-26T00:03:19.800 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:03:19.800 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:03:19.801 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:03:19.801 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:03:19.801 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:03:19.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:03:19.803 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-26T00:03:19.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-26T00:03:19.805 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-26T00:03:19.807 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-26T00:03:19.807 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-26T00:03:19.807 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-26T00:03:19.809 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:03:19.809 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-26T00:03:19.809 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-26T00:03:19.809 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:03:19.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-26T00:03:19.812 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-26T00:03:19.812 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-26T00:03:19.814 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-26T00:03:19.814 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:03:19.814 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:03:19.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-26T00:03:19.815 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-26T00:03:19.815 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-26T00:03:19.815 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-26T00:03:19.817 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-26T00:03:19.817 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:03:19.817 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:03:19.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61636 2024-03-26T00:03:19.820 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-26T00:03:19.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-26T00:03:19.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-26T00:03:19.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-26T00:03:19.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_erasure_appends td/osd-scrub-repair 2024-03-26T00:03:19.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5746: TEST_corrupt_scrub_erasure_appends: corrupt_scrub_erasure td/osd-scrub-repair false 2024-03-26T00:03:19.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3596: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2024-03-26T00:03:19.821 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3597: corrupt_scrub_erasure: local allow_overwrites=false 2024-03-26T00:03:19.821 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3598: corrupt_scrub_erasure: local poolname=ecpool 2024-03-26T00:03:19.821 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3599: corrupt_scrub_erasure: local total_objs=7 2024-03-26T00:03:19.821 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3601: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2024-03-26T00:03:19.821 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-26T00:03:19.821 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-26T00:03:19.821 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-26T00:03:19.821 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-26T00:03:19.821 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-26T00:03:19.821 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-26T00:03:20.309 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-26T00:03:20.309 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:03:20.309 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:03:20.309 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:03:20.309 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:03:20.309 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:03:20.310 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:03:20.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-26T00:03:20.525 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-26T00:03:20.527 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-26T00:03:20.527 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-26T00:03:20.527 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-26T00:03:20.527 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-26T00:03:20.527 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-26T00:03:20.528 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-26T00:03:20.528 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-26T00:03:20.528 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-26T00:03:20.528 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-26T00:03:20.529 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:03:20.529 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:03:20.529 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-26T00:03:20.529 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-26T00:03:20.529 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get fsid 2024-03-26T00:03:20.645 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-26T00:03:20.645 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-26T00:03:20.645 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-26T00:03:20.645 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-26T00:03:20.646 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-26T00:03:20.646 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-26T00:03:20.646 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-26T00:03:20.647 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-26T00:03:20.647 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-26T00:03:20.647 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:03:20.647 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:03:20.647 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-26T00:03:20.647 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-26T00:03:20.647 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get mon_host 2024-03-26T00:03:20.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3602: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2024-03-26T00:03:20.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-26T00:03:20.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-26T00:03:20.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-26T00:03:20.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-26T00:03:20.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-26T00:03:20.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-26T00:03:21.643 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-26T00:03:21.643 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:03:21.643 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:03:21.644 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:03:21.644 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:03:21.644 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:03:21.644 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:03:21.645 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-26T00:03:21.647 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-26T00:03:21.682 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: seq 0 2 2024-03-26T00:03:21.685 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-03-26T00:03:21.685 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 0 2024-03-26T00:03:21.686 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-26T00:03:21.686 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-26T00:03:21.686 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-26T00:03:21.686 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-26T00:03:21.686 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:03:21.686 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:03:21.686 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:03:21.686 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:03:21.686 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:03:21.686 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-26T00:03:21.686 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-26T00:03:21.686 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-26T00:03:21.686 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-26T00:03:21.687 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:03:21.687 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-26T00:03:21.687 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:03:21.687 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:03:21.687 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:03:21.687 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:03:21.687 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:03:21.687 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:03:21.687 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:03:21.687 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:03:21.687 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-26T00:03:21.687 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-26T00:03:21.687 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:03:21.687 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:03:21.687 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:03:21.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:03:21.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:03:21.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:03:21.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-26T00:03:21.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-26T00:03:21.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-26T00:03:21.688 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-26T00:03:21.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0eae8a05-f177-416a-8c75-3129f39d10de 2024-03-26T00:03:21.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0eae8a05-f177-416a-8c75-3129f39d10de' 2024-03-26T00:03:21.689 INFO:tasks.workunit.client.0.smithi129.stdout:add osd0 0eae8a05-f177-416a-8c75-3129f39d10de 2024-03-26T00:03:21.689 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-26T00:03:21.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDJEAJm5eE1KhAAPdE6AtXgg1wMHStWz+yaXA== 2024-03-26T00:03:21.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDJEAJm5eE1KhAAPdE6AtXgg1wMHStWz+yaXA=="}' 2024-03-26T00:03:21.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0eae8a05-f177-416a-8c75-3129f39d10de -i td/osd-scrub-repair/0/new.json 2024-03-26T00:03:21.973 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-26T00:03:21.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-26T00:03:21.992 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDJEAJm5eE1KhAAPdE6AtXgg1wMHStWz+yaXA== --osd-uuid 0eae8a05-f177-416a-8c75-3129f39d10de 2024-03-26T00:03:22.026 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:03:22.022+0000 7fc086d1e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:03:22.040 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:03:22.038+0000 7fc086d1e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:03:22.041 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:03:22.038+0000 7fc086d1e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:03:22.042 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:03:22.038+0000 7fc086d1e7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-26T00:03:22.042 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:03:22.038+0000 7fc086d1e7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-26T00:03:22.042 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:03:22.038+0000 7fc086d1e7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-26T00:03:22.132 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:03:22.130+0000 7fc086d1e7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-26T00:03:28.543 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-26T00:03:28.544 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-26T00:03:28.545 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd0 key to auth repository 2024-03-26T00:03:28.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-26T00:03:28.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-26T00:03:28.942 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-26T00:03:28.942 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.0 2024-03-26T00:03:28.943 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:03:28.943 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-26T00:03:28.943 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-26T00:03:28.943 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-26T00:03:28.974 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:03:28.970+0000 7f9c1b6287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:03:29.016 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:03:29.014+0000 7f9c1b6287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:03:29.017 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:03:29.014+0000 7f9c1b6287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:03:29.285 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-26T00:03:29.285 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:03:29.285 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-26T00:03:29.285 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:03:29.285 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:03:29.285 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:03:29.286 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-26T00:03:29.286 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:03:29.286 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:03:29.287 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:03:29.569 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:03:29.566+0000 7f9c1b6287c0 -1 Falling back to public interface 2024-03-26T00:03:29.617 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:03:30.619 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:03:30.619 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:03:30.619 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-26T00:03:30.620 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:03:30.620 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:03:30.620 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:03:30.936 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:03:31.937 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:03:31.938 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:03:31.938 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:03:31.938 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-26T00:03:31.938 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:03:31.939 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:03:32.220 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:03:32.218+0000 7f9c1b6287c0 -1 osd.0 0 log_to_monitors true 2024-03-26T00:03:32.248 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:03:33.250 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:03:33.250 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:03:33.250 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-26T00:03:33.251 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:03:33.251 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:03:33.251 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:03:33.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:03:34.566 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:03:34.566 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:03:34.566 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-26T00:03:34.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-26T00:03:34.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:03:34.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:03:34.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:03:35.901 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:03:35.901 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:03:35.902 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-26T00:03:35.902 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-26T00:03:35.902 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:03:35.902 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:03:36.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:03:37.282 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:03:37.282 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:03:37.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-26T00:03:37.283 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-26T00:03:37.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:03:37.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:03:37.591 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:03:38.593 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:03:38.593 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:03:38.594 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-26T00:03:38.594 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-26T00:03:38.594 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:03:38.594 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:03:38.905 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:03:39.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:03:39.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:03:39.908 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-26T00:03:39.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-26T00:03:39.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:03:39.909 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:03:40.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:03:41.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:03:41.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:03:41.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-26T00:03:41.300 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-26T00:03:41.300 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:03:41.300 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:03:41.547 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:03:42.549 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:03:42.549 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:03:42.549 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-26T00:03:42.549 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-26T00:03:42.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:03:42.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:03:42.900 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:03:43.902 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:03:43.902 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:03:43.902 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-26T00:03:43.903 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-26T00:03:43.903 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:03:43.903 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:03:44.292 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:03:45.295 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:03:45.295 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:03:45.295 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-26T00:03:45.296 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-26T00:03:45.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:03:45.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:03:45.614 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:03:46.616 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:03:46.616 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:03:46.617 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-26T00:03:46.617 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-26T00:03:46.617 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:03:46.617 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:03:47.091 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:03:48.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:03:48.094 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-26T00:03:48.094 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:03:48.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-26T00:03:48.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:03:48.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:03:48.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:03:49.669 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:03:49.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:03:49.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-26T00:03:49.670 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-26T00:03:49.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:03:49.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:03:50.491 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:03:51.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:03:51.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:03:51.493 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-26T00:03:51.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-26T00:03:51.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:03:51.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:03:52.084 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:03:53.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:03:53.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:03:53.087 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-26T00:03:53.087 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-26T00:03:53.087 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:03:53.087 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:03:53.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:03:54.819 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:03:54.819 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:03:54.819 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-26T00:03:54.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-26T00:03:54.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:03:54.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:03:55.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:03:56.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:03:56.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:03:56.189 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-26T00:03:56.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-26T00:03:56.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:03:56.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:03:56.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:03:57.718 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:03:57.718 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:03:57.719 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-26T00:03:57.719 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-26T00:03:57.719 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:03:57.719 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:03:58.030 INFO:tasks.workunit.client.0.smithi129.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3979197242,v1:127.0.0.1:6803/3979197242] [v2:127.0.0.1:6804/3979197242,v1:127.0.0.1:6805/3979197242] exists,up 0eae8a05-f177-416a-8c75-3129f39d10de 2024-03-26T00:03:58.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:03:58.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:03:58.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:03:58.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-03-26T00:03:58.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 1 2024-03-26T00:03:58.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-26T00:03:58.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-26T00:03:58.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-26T00:03:58.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-26T00:03:58.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:03:58.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:03:58.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:03:58.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:03:58.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:03:58.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:03:58.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:03:58.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-26T00:03:58.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-26T00:03:58.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:03:58.034 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-26T00:03:58.034 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:03:58.034 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:03:58.034 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:03:58.034 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:03:58.034 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:03:58.035 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:03:58.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:03:58.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:03:58.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-26T00:03:58.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-26T00:03:58.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:03:58.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:03:58.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:03:58.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:03:58.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:03:58.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:03:58.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-26T00:03:58.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-26T00:03:58.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:03:58.037 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-26T00:03:58.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c51d3969-0ab9-40e5-91ca-849223a6c0c5 2024-03-26T00:03:58.037 INFO:tasks.workunit.client.0.smithi129.stdout:add osd1 c51d3969-0ab9-40e5-91ca-849223a6c0c5 2024-03-26T00:03:58.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 c51d3969-0ab9-40e5-91ca-849223a6c0c5' 2024-03-26T00:03:58.038 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-26T00:03:58.052 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDuEAJmC9dJAxAAJ4H+sIJbZhgRrH4cumrzZQ== 2024-03-26T00:03:58.053 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDuEAJmC9dJAxAAJ4H+sIJbZhgRrH4cumrzZQ=="}' 2024-03-26T00:03:58.053 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c51d3969-0ab9-40e5-91ca-849223a6c0c5 -i td/osd-scrub-repair/1/new.json 2024-03-26T00:03:58.425 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-26T00:03:58.444 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-26T00:03:58.445 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDuEAJmC9dJAxAAJ4H+sIJbZhgRrH4cumrzZQ== --osd-uuid c51d3969-0ab9-40e5-91ca-849223a6c0c5 2024-03-26T00:03:58.482 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:03:58.481+0000 7f82e8c437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:03:58.527 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:03:58.529+0000 7f82e8c437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:03:58.529 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:03:58.529+0000 7f82e8c437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:03:58.529 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:03:58.529+0000 7f82e8c437c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-26T00:03:58.529 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:03:58.529+0000 7f82e8c437c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-26T00:03:58.530 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:03:58.529+0000 7f82e8c437c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-26T00:03:58.643 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:03:58.645+0000 7f82e8c437c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-26T00:04:05.404 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-26T00:04:05.404 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-26T00:04:05.405 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd1 key to auth repository 2024-03-26T00:04:05.405 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-26T00:04:05.405 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-26T00:04:05.787 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.1 2024-03-26T00:04:05.788 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-26T00:04:05.788 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:04:05.788 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-26T00:04:05.788 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-26T00:04:05.788 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-26T00:04:05.820 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:04:05.821+0000 7fc9983297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:04:05.849 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:04:05.849+0000 7fc9983297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:04:05.851 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:04:05.853+0000 7fc9983297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:04:06.122 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-26T00:04:06.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:04:06.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:04:06.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:04:06.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:04:06.123 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-26T00:04:06.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:04:06.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:04:06.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:04:06.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:04:06.397 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:04:06.397+0000 7fc9983297c0 -1 Falling back to public interface 2024-03-26T00:04:06.486 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:04:07.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:04:07.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:04:07.489 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-26T00:04:07.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:04:07.490 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:04:07.490 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:04:07.808 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:04:08.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:04:08.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:04:08.811 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-26T00:04:08.811 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:04:08.811 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:04:08.811 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:04:09.086 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:04:09.085+0000 7fc9983297c0 -1 osd.1 0 log_to_monitors true 2024-03-26T00:04:09.152 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:04:10.154 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:04:10.154 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:04:10.154 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-26T00:04:10.155 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:04:10.155 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:04:10.155 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:04:10.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:04:11.519 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:04:11.519 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:04:11.519 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-26T00:04:11.520 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-26T00:04:11.520 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:04:11.520 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:04:11.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:04:12.981 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:04:12.981 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:04:12.981 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-26T00:04:12.982 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-26T00:04:12.982 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:04:12.982 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:04:13.304 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:04:14.306 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:04:14.306 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:04:14.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-26T00:04:14.307 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-26T00:04:14.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:04:14.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:04:14.636 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:04:15.638 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:04:15.638 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:04:15.638 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-26T00:04:15.639 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-26T00:04:15.639 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:04:15.639 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:04:15.993 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:04:16.996 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:04:16.996 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:04:16.996 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-26T00:04:16.996 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-26T00:04:16.997 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:04:16.997 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:04:17.320 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:04:18.322 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:04:18.322 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:04:18.322 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-26T00:04:18.322 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-26T00:04:18.323 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:04:18.323 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:04:18.641 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:04:19.643 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:04:19.643 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:04:19.643 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-26T00:04:19.643 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-26T00:04:19.644 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:04:19.644 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:04:20.103 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:04:21.105 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:04:21.106 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:04:21.106 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-26T00:04:21.106 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-26T00:04:21.106 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:04:21.106 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:04:21.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:04:22.506 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:04:22.506 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:04:22.507 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-26T00:04:22.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-26T00:04:22.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:04:22.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:04:22.945 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:04:23.946 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:04:23.947 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:04:23.947 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-26T00:04:23.947 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-26T00:04:23.947 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:04:23.947 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:04:24.823 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:04:25.825 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:04:25.825 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:04:25.826 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-26T00:04:25.826 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-26T00:04:25.826 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:04:25.826 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:04:26.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:04:27.487 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:04:27.487 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:04:27.487 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-26T00:04:27.487 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-26T00:04:27.488 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:04:27.488 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:04:28.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:04:29.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:04:29.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:04:29.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-26T00:04:29.138 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-26T00:04:29.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:04:29.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:04:29.883 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:04:30.885 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:04:30.886 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:04:30.886 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-26T00:04:30.886 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-26T00:04:30.886 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:04:30.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:04:31.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:04:32.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:04:32.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:04:32.516 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-26T00:04:32.516 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-26T00:04:32.516 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:04:32.516 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:04:33.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:04:34.255 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:04:34.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:04:34.256 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-26T00:04:34.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-26T00:04:34.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:04:34.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:04:34.605 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:04:35.606 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:04:35.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:04:35.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-26T00:04:35.607 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-26T00:04:35.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:04:35.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:04:35.925 INFO:tasks.workunit.client.0.smithi129.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2609455735,v1:127.0.0.1:6811/2609455735] [v2:127.0.0.1:6812/2609455735,v1:127.0.0.1:6813/2609455735] exists,up c51d3969-0ab9-40e5-91ca-849223a6c0c5 2024-03-26T00:04:35.928 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:04:35.928 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:04:35.928 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:04:35.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-03-26T00:04:35.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 2 2024-03-26T00:04:35.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-26T00:04:35.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-26T00:04:35.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-26T00:04:35.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-26T00:04:35.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-26T00:04:35.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:04:35.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:04:35.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:04:35.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:04:35.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-26T00:04:35.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-26T00:04:35.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-26T00:04:35.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-26T00:04:35.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:04:35.930 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-26T00:04:35.930 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:04:35.930 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:04:35.930 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:04:35.930 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:04:35.930 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:04:35.930 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:04:35.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:04:35.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:04:35.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-26T00:04:35.931 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-26T00:04:35.931 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:04:35.931 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:04:35.931 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:04:35.931 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:04:35.931 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:04:35.931 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:04:35.931 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-26T00:04:35.931 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-26T00:04:35.931 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-26T00:04:35.932 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-26T00:04:35.932 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=27be98cf-bbe7-4a2e-99cb-ddbe940370a6 2024-03-26T00:04:35.932 INFO:tasks.workunit.client.0.smithi129.stdout:add osd2 27be98cf-bbe7-4a2e-99cb-ddbe940370a6 2024-03-26T00:04:35.932 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 27be98cf-bbe7-4a2e-99cb-ddbe940370a6' 2024-03-26T00:04:35.933 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-26T00:04:35.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQATEQJm/f3rOBAA61+Hz6IIx/KfqME0LLidkw== 2024-03-26T00:04:35.953 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQATEQJm/f3rOBAA61+Hz6IIx/KfqME0LLidkw=="}' 2024-03-26T00:04:35.953 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 27be98cf-bbe7-4a2e-99cb-ddbe940370a6 -i td/osd-scrub-repair/2/new.json 2024-03-26T00:04:36.417 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-26T00:04:36.439 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-26T00:04:36.440 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQATEQJm/f3rOBAA61+Hz6IIx/KfqME0LLidkw== --osd-uuid 27be98cf-bbe7-4a2e-99cb-ddbe940370a6 2024-03-26T00:04:36.476 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:04:36.476+0000 7f831f2b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:04:36.518 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:04:36.520+0000 7f831f2b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:04:36.520 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:04:36.520+0000 7f831f2b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:04:36.520 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:04:36.520+0000 7f831f2b87c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-26T00:04:36.520 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:04:36.520+0000 7f831f2b87c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-26T00:04:36.520 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:04:36.520+0000 7f831f2b87c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-26T00:04:36.595 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:04:36.596+0000 7f831f2b87c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-26T00:04:43.288 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-26T00:04:43.288 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-26T00:04:43.289 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd2 key to auth repository 2024-03-26T00:04:43.289 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-26T00:04:43.289 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-26T00:04:43.669 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-26T00:04:43.670 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.2 2024-03-26T00:04:43.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-26T00:04:43.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-26T00:04:43.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:04:43.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-26T00:04:43.701 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:04:43.700+0000 7fa8b93b67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:04:43.703 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:04:43.704+0000 7fa8b93b67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:04:43.704 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:04:43.704+0000 7fa8b93b67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:04:43.981 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-26T00:04:43.981 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:04:43.981 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-26T00:04:43.981 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:04:43.982 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-26T00:04:43.982 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:04:43.982 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:04:43.982 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:04:43.982 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:04:43.982 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:04:44.281 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:04:44.280+0000 7fa8b93b67c0 -1 Falling back to public interface 2024-03-26T00:04:44.336 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:04:45.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:04:45.338 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:04:45.338 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-26T00:04:45.338 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:04:45.339 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:04:45.339 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:04:45.643 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:04:46.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:04:46.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:04:46.645 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-26T00:04:46.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:04:46.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:04:46.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:04:46.987 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:04:46.988+0000 7fa8b93b67c0 -1 osd.2 0 log_to_monitors true 2024-03-26T00:04:46.999 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:04:48.001 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:04:48.001 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:04:48.002 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-26T00:04:48.002 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:04:48.002 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:04:48.002 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:04:48.323 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:04:49.325 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:04:49.325 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:04:49.325 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-26T00:04:49.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-26T00:04:49.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:04:49.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:04:49.638 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:04:50.640 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:04:50.640 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-26T00:04:50.641 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:04:50.641 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-26T00:04:50.641 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:04:50.641 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:04:50.960 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:04:51.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:04:51.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:04:51.962 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-26T00:04:51.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-26T00:04:51.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:04:51.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:04:52.293 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:04:53.295 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:04:53.295 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:04:53.296 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-26T00:04:53.296 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-26T00:04:53.296 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:04:53.296 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:04:53.621 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:04:54.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:04:54.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:04:54.623 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-26T00:04:54.624 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-26T00:04:54.624 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:04:54.624 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:04:55.061 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:04:56.064 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:04:56.066 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:04:56.067 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-26T00:04:56.068 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-26T00:04:56.068 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:04:56.068 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:04:56.383 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:04:57.385 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:04:57.385 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:04:57.386 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-26T00:04:57.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-26T00:04:57.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:04:57.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:04:57.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:04:58.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:04:58.733 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:04:58.733 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-26T00:04:58.733 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-26T00:04:58.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:04:58.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:04:59.282 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:05:00.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:05:00.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:05:00.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-26T00:05:00.285 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-26T00:05:00.285 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:05:00.285 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:05:00.606 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:05:01.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:05:01.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:05:01.608 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-26T00:05:01.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-26T00:05:01.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:05:01.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:05:02.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:05:03.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:05:03.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:05:03.391 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-26T00:05:03.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-26T00:05:03.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:05:03.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:05:04.055 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:05:05.057 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:05:05.057 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:05:05.058 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-26T00:05:05.058 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-26T00:05:05.058 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:05:05.058 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:05:05.473 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:05:06.475 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:05:06.475 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:05:06.475 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-26T00:05:06.476 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-26T00:05:06.476 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:05:06.476 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:05:07.053 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:05:08.055 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:05:08.055 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:05:08.055 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-26T00:05:08.055 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-26T00:05:08.056 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:05:08.056 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:05:08.775 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:05:09.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:05:09.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:05:09.778 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-26T00:05:09.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-26T00:05:09.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:05:09.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:05:10.354 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:05:11.356 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:05:11.356 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:05:11.356 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-26T00:05:11.357 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-26T00:05:11.357 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:05:11.357 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:05:11.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:05:12.705 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:05:12.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:05:12.706 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-26T00:05:12.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-26T00:05:12.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:05:12.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:05:13.060 INFO:tasks.workunit.client.0.smithi129.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3571885439,v1:127.0.0.1:6819/3571885439] [v2:127.0.0.1:6820/3571885439,v1:127.0.0.1:6821/3571885439] exists,up 27be98cf-bbe7-4a2e-99cb-ddbe940370a6 2024-03-26T00:05:13.061 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:05:13.061 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:05:13.061 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:05:13.061 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3606: corrupt_scrub_erasure: create_rbd_pool 2024-03-26T00:05:13.061 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-26T00:05:13.415 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'rbd' does not exist 2024-03-26T00:05:13.432 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-26T00:05:13.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-26T00:05:13.870 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'rbd' created 2024-03-26T00:05:13.893 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-26T00:05:14.895 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-26T00:05:17.891 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3607: corrupt_scrub_erasure: create_pool foo 1 2024-03-26T00:05:17.891 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-03-26T00:05:18.370 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'foo' created 2024-03-26T00:05:18.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-26T00:05:19.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3609: corrupt_scrub_erasure: create_ec_pool ecpool false k=2 m=1 stripe_unit=2K --force 2024-03-26T00:05:19.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-26T00:05:19.393 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-26T00:05:19.393 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-03-26T00:05:19.393 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-26T00:05:19.393 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2024-03-26T00:05:19.871 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-26T00:05:19.871 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-26T00:05:20.573 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'ecpool' created 2024-03-26T00:05:20.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-26T00:05:21.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-03-26T00:05:21.600 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-26T00:05:21.600 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:05:21.600 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:05:21.600 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:05:21.600 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:05:21.601 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:05:21.601 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:05:21.602 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:05:21.602 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:05:21.602 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:05:21.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:05:21.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:05:21.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:05:21.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:05:21.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:05:21.731 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:05:22.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:05:22.048 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:05:22.048 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-26T00:05:22.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:05:22.049 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:05:22.049 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:05:22.301 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836503 2024-03-26T00:05:22.301 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836503 2024-03-26T00:05:22.301 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503' 2024-03-26T00:05:22.302 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:05:22.302 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:05:22.446 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672979 2024-03-26T00:05:22.447 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672979 2024-03-26T00:05:22.447 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672979' 2024-03-26T00:05:22.447 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:05:22.447 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:05:22.594 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509445 2024-03-26T00:05:22.595 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509445 2024-03-26T00:05:22.595 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672979 2-64424509445' 2024-03-26T00:05:22.595 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:05:22.595 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:05:22.595 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836503 2024-03-26T00:05:22.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:05:22.597 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836503 2024-03-26T00:05:22.597 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:05:22.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836503 2024-03-26T00:05:22.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-03-26T00:05:22.598 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 21474836503 2024-03-26T00:05:22.598 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:05:22.945 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836503 2024-03-26T00:05:22.946 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:05:23.947 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:05:23.948 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:05:24.293 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836503 2024-03-26T00:05:24.293 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:05:25.295 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-26T00:05:25.296 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:05:25.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836503 2024-03-26T00:05:25.616 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:05:25.616 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672979 2024-03-26T00:05:25.616 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:05:25.618 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:05:25.619 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672979 2024-03-26T00:05:25.619 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:05:25.620 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672979 2024-03-26T00:05:25.620 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-03-26T00:05:25.620 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 42949672979 2024-03-26T00:05:25.621 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:05:25.916 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672979 -lt 42949672979 2024-03-26T00:05:25.916 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:05:25.917 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509445 2024-03-26T00:05:25.917 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:05:25.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:05:25.920 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509445 2024-03-26T00:05:25.920 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:05:25.921 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509445 2024-03-26T00:05:25.921 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 64424509445 2024-03-26T00:05:25.921 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-03-26T00:05:25.922 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:05:26.295 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509446 -lt 64424509445 2024-03-26T00:05:26.296 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:05:26.296 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:05:26.297 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:05:26.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:05:26.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:05:26.804 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:05:26.804 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:05:26.804 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:05:26.804 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:05:26.804 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:05:26.804 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:05:27.131 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:05:27.131 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:05:27.132 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:05:27.132 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:05:27.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:05:27.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:05:27.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:05:27.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-26T00:05:27.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3610: corrupt_scrub_erasure: wait_for_clean 2024-03-26T00:05:27.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:05:27.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:05:27.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:05:27.506 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:05:27.506 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:05:27.506 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:05:27.506 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:05:27.507 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:05:27.507 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:05:27.612 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:05:27.612 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:05:27.612 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:05:27.612 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:05:27.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:05:27.613 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:05:27.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:05:27.907 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:05:27.907 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-26T00:05:27.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:05:27.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:05:27.907 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:05:28.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836506 2024-03-26T00:05:28.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836506 2024-03-26T00:05:28.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506' 2024-03-26T00:05:28.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:05:28.097 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:05:28.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672981 2024-03-26T00:05:28.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672981 2024-03-26T00:05:28.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-42949672981' 2024-03-26T00:05:28.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:05:28.244 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:05:28.384 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509448 2024-03-26T00:05:28.384 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509448 2024-03-26T00:05:28.384 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-42949672981 2-64424509448' 2024-03-26T00:05:28.384 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:05:28.385 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836506 2024-03-26T00:05:28.385 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:05:28.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:05:28.386 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836506 2024-03-26T00:05:28.387 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:05:28.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836506 2024-03-26T00:05:28.388 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 21474836506 2024-03-26T00:05:28.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-03-26T00:05:28.388 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:05:28.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836506 2024-03-26T00:05:28.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:05:28.671 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672981 2024-03-26T00:05:28.671 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:05:28.673 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:05:28.674 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672981 2024-03-26T00:05:28.674 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:05:28.675 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672981 2024-03-26T00:05:28.675 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 42949672981 2024-03-26T00:05:28.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-03-26T00:05:28.676 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:05:28.999 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672980 -lt 42949672981 2024-03-26T00:05:28.999 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:05:30.001 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:05:30.001 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:05:30.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672980 -lt 42949672981 2024-03-26T00:05:30.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:05:31.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-26T00:05:31.311 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:05:31.614 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672981 -lt 42949672981 2024-03-26T00:05:31.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:05:31.616 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509448 2024-03-26T00:05:31.616 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:05:31.618 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:05:31.619 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509448 2024-03-26T00:05:31.619 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:05:31.620 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509448 2024-03-26T00:05:31.620 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 64424509448 2024-03-26T00:05:31.620 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-03-26T00:05:31.621 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:05:31.944 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509448 -lt 64424509448 2024-03-26T00:05:31.945 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:05:31.945 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:05:31.945 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:05:32.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:05:32.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:05:32.350 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:05:32.350 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:05:32.350 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:05:32.350 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:05:32.350 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:05:32.350 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:05:32.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:05:32.660 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:05:32.661 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:05:32.661 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:05:33.044 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:05:33.044 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:05:33.044 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:05:33.044 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: seq 1 7 2024-03-26T00:05:33.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-26T00:05:33.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ1 2024-03-26T00:05:33.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2024-03-26T00:05:33.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-26T00:05:33.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-26T00:05:33.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ1 2024-03-26T00:05:33.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-26T00:05:33.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-26T00:05:33.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-26T00:05:33.466 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-26T00:05:33.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-26T00:05:34.044 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-26T00:05:34.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-26T00:05:34.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-26T00:05:34.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:05:34.350 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 1 % 2 2024-03-26T00:05:34.354 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-03-26T00:05:34.354 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-26T00:05:34.354 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3621: corrupt_scrub_erasure: local payload=UVWXYZZZ 2024-03-26T00:05:34.354 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3622: corrupt_scrub_erasure: echo UVWXYZZZ 2024-03-26T00:05:34.354 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3623: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-26T00:05:34.354 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:05:34.354 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:05:34.354 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-26T00:05:34.355 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:05:34.355 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-26T00:05:34.355 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:05:34.355 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:05:34.355 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-26T00:05:34.355 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:05:34.355 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-26T00:05:34.355 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:05:34.355 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:05:34.355 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:05:34.355 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:05:34.356 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:05:35.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:05:35.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-26T00:05:35.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:05:35.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:05:35.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:05:35.668 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:05:35.668 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:05:35.668 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-26T00:05:37.796 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-26T00:05:37.796 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:05:37.796 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:05:37.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-26T00:05:37.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:05:37.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:05:37.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:05:37.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:05:37.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:05:37.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:05:37.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:05:37.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:05:37.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:05:37.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:05:37.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:05:37.798 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:05:37.798 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:05:37.798 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:05:37.799 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:05:37.799 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:05:37.799 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:05:37.799 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:05:37.799 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:05:37.799 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:05:37.799 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:05:37.799 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:05:37.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:05:37.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:05:37.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:05:37.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:05:37.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:05:37.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:05:37.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:05:37.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-26T00:05:37.800 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.1 2024-03-26T00:05:37.801 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:05:37.801 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-26T00:05:37.801 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-26T00:05:37.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:05:37.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:05:37.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:05:37.833 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:05:37.831+0000 7f1324c177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:05:37.836 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:05:37.835+0000 7f1324c177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:05:37.837 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:05:37.835+0000 7f1324c177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:05:38.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-26T00:05:38.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:05:38.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:05:38.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:05:38.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:05:38.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:05:38.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:05:38.113 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:05:38.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:05:38.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:05:38.376 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:05:38.375+0000 7f1324c177c0 -1 Falling back to public interface 2024-03-26T00:05:38.502 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:05:39.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:05:39.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:05:39.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:05:39.504 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:05:39.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:05:39.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:05:39.809 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:05:40.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:05:40.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:05:40.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:05:40.810 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:05:40.811 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:05:40.811 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:05:40.989 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:05:40.987+0000 7f1324c177c0 -1 osd.1 36 log_to_monitors true 2024-03-26T00:05:41.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:05:42.242 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:05:42.242 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:05:42.242 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:05:42.242 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:05:42.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:05:42.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:05:42.686 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1 up in weight 1 up_from 41 up_thru 41 down_at 37 last_clean_interval [10,36) [v2:127.0.0.1:6810/2770351329,v1:127.0.0.1:6811/2770351329] [v2:127.0.0.1:6812/2770351329,v1:127.0.0.1:6813/2770351329] exists,up c51d3969-0ab9-40e5-91ca-849223a6c0c5 2024-03-26T00:05:42.686 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:05:42.686 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:05:42.686 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:05:42.686 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:05:42.686 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:05:42.686 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:05:42.687 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:05:42.688 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:05:42.688 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:05:42.688 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:05:42.688 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:05:42.688 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:05:42.688 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:05:42.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:05:42.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:05:42.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:05:42.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:05:42.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:05:42.798 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:05:43.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:05:43.240 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:05:43.240 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-26T00:05:43.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:05:43.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:05:43.240 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:05:43.408 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836510 2024-03-26T00:05:43.408 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836510 2024-03-26T00:05:43.408 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510' 2024-03-26T00:05:43.408 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:05:43.409 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:05:43.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659138 2024-03-26T00:05:43.551 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659138 2024-03-26T00:05:43.551 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510 1-176093659138' 2024-03-26T00:05:43.551 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:05:43.551 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:05:43.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509452 2024-03-26T00:05:43.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509452 2024-03-26T00:05:43.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510 1-176093659138 2-64424509452' 2024-03-26T00:05:43.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:05:43.698 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836510 2024-03-26T00:05:43.698 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:05:43.699 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:05:43.699 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836510 2024-03-26T00:05:43.699 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:05:43.701 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836510 2024-03-26T00:05:43.701 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-03-26T00:05:43.701 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 21474836510 2024-03-26T00:05:43.702 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:05:44.009 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836510 2024-03-26T00:05:44.009 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:05:45.011 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:05:45.011 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:05:45.315 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836510 -lt 21474836510 2024-03-26T00:05:45.315 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:05:45.316 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-176093659138 2024-03-26T00:05:45.316 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:05:45.317 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:05:45.317 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-176093659138 2024-03-26T00:05:45.317 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:05:45.318 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659138 2024-03-26T00:05:45.318 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 176093659138' 2024-03-26T00:05:45.319 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 176093659138 2024-03-26T00:05:45.319 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:05:45.666 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659138 -lt 176093659138 2024-03-26T00:05:45.666 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:05:45.667 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:05:45.667 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509452 2024-03-26T00:05:45.668 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:05:45.670 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509452 2024-03-26T00:05:45.670 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:05:45.672 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509452 2024-03-26T00:05:45.672 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-03-26T00:05:45.672 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 64424509452 2024-03-26T00:05:45.672 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:05:46.016 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509452 -lt 64424509452 2024-03-26T00:05:46.017 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:05:46.017 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:05:46.017 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:05:46.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:05:46.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:05:46.420 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:05:46.420 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:05:46.420 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:05:46.420 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:05:46.421 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:05:46.421 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:05:46.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:05:46.761 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:05:46.761 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:05:46.762 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:05:47.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:05:47.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:05:47.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:05:47.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-26T00:05:47.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ2 2024-03-26T00:05:47.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2024-03-26T00:05:47.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-26T00:05:47.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-26T00:05:47.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ2 2024-03-26T00:05:47.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-26T00:05:47.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-26T00:05:47.133 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-26T00:05:47.571 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-26T00:05:47.589 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-26T00:05:48.132 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-26T00:05:48.154 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-26T00:05:48.154 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-26T00:05:48.154 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:05:48.557 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 2 % 2 2024-03-26T00:05:48.558 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-03-26T00:05:48.558 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-26T00:05:48.558 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3628: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2024-03-26T00:05:48.560 INFO:tasks.workunit.client.0.smithi129.stderr:1+0 records in 2024-03-26T00:05:48.560 INFO:tasks.workunit.client.0.smithi129.stderr:1+0 records out 2024-03-26T00:05:48.560 INFO:tasks.workunit.client.0.smithi129.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 0.000162744 s, 12.6 MB/s 2024-03-26T00:05:48.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-26T00:05:48.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:05:48.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:05:48.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-26T00:05:48.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:05:48.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-26T00:05:48.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:05:48.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:05:48.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-26T00:05:48.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:05:48.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-26T00:05:48.561 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:05:48.561 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:05:48.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:05:48.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:05:48.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:05:48.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:05:48.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-26T00:05:48.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:05:48.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:05:48.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:05:48.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:05:48.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:05:48.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-26T00:05:51.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-26T00:05:51.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:05:51.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:05:51.610 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-26T00:05:51.610 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:05:51.610 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:05:51.610 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:05:51.610 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:05:51.610 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:05:51.610 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:05:51.610 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-26T00:05:51.610 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-26T00:05:51.610 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:05:51.610 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:05:51.610 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:05:51.611 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:05:51.611 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:05:51.611 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:05:51.611 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:05:51.611 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:05:51.611 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:05:51.612 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:05:51.612 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:05:51.612 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:05:51.612 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:05:51.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:05:51.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:05:51.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:05:51.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:05:51.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:05:51.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:05:51.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:05:51.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-26T00:05:51.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-26T00:05:51.613 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.0 2024-03-26T00:05:51.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:05:51.614 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-26T00:05:51.614 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-26T00:05:51.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:05:51.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:05:51.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:05:51.646 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:05:51.643+0000 7f061d50e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:05:51.649 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:05:51.647+0000 7f061d50e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:05:51.650 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:05:51.651+0000 7f061d50e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:05:51.936 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-26T00:05:51.936 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:05:51.936 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-26T00:05:51.936 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:05:51.936 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:05:51.936 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:05:51.936 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:05:51.936 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:05:51.936 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:05:51.936 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:05:52.185 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:05:52.183+0000 7f061d50e7c0 -1 Falling back to public interface 2024-03-26T00:05:52.292 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:05:53.294 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:05:53.294 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:05:53.294 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:05:53.294 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:05:53.295 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:05:53.295 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:05:53.612 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:05:54.614 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:05:54.614 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:05:54.614 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:05:54.614 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:05:54.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:05:54.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:05:54.907 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:05:54.907+0000 7f061d50e7c0 -1 osd.0 45 log_to_monitors true 2024-03-26T00:05:54.942 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:05:55.944 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:05:55.944 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:05:55.944 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:05:55.944 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:05:55.944 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:05:55.944 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:05:56.401 INFO:tasks.workunit.client.0.smithi129.stderr:osd.0 up in weight 1 up_from 50 up_thru 50 down_at 46 last_clean_interval [5,45) [v2:127.0.0.1:6802/2202352822,v1:127.0.0.1:6803/2202352822] [v2:127.0.0.1:6804/2202352822,v1:127.0.0.1:6805/2202352822] exists,up 0eae8a05-f177-416a-8c75-3129f39d10de 2024-03-26T00:05:56.401 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:05:56.402 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:05:56.402 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:05:56.402 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:05:56.402 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:05:56.402 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:05:56.402 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:05:56.403 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:05:56.403 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:05:56.403 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:05:56.403 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:05:56.403 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:05:56.403 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:05:56.514 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:05:56.514 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:05:56.514 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:05:56.514 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:05:56.514 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:05:56.514 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:05:56.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:05:56.963 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:05:56.963 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-26T00:05:56.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:05:56.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:05:56.963 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:05:57.107 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364802 2024-03-26T00:05:57.107 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364802 2024-03-26T00:05:57.107 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-214748364802' 2024-03-26T00:05:57.107 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:05:57.107 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:05:57.268 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659142 2024-03-26T00:05:57.268 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659142 2024-03-26T00:05:57.268 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-214748364802 1-176093659142' 2024-03-26T00:05:57.269 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:05:57.269 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:05:57.410 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509455 2024-03-26T00:05:57.410 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509455 2024-03-26T00:05:57.410 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-214748364802 1-176093659142 2-64424509455' 2024-03-26T00:05:57.411 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:05:57.411 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-214748364802 2024-03-26T00:05:57.411 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:05:57.412 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:05:57.413 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:05:57.413 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-214748364802 2024-03-26T00:05:57.414 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364802 2024-03-26T00:05:57.414 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 214748364802' 2024-03-26T00:05:57.414 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 214748364802 2024-03-26T00:05:57.414 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:05:57.721 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 214748364802 2024-03-26T00:05:57.721 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:05:58.722 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:05:58.723 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:05:59.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364802 -lt 214748364802 2024-03-26T00:05:59.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:05:59.082 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-176093659142 2024-03-26T00:05:59.082 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:05:59.083 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:05:59.084 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-176093659142 2024-03-26T00:05:59.084 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:05:59.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659142 2024-03-26T00:05:59.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 176093659142' 2024-03-26T00:05:59.086 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 176093659142 2024-03-26T00:05:59.086 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:05:59.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659142 -lt 176093659142 2024-03-26T00:05:59.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:05:59.390 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509455 2024-03-26T00:05:59.390 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:05:59.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:05:59.392 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509455 2024-03-26T00:05:59.392 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:05:59.393 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509455 2024-03-26T00:05:59.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-03-26T00:05:59.394 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 64424509455 2024-03-26T00:05:59.394 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:05:59.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509455 2024-03-26T00:05:59.716 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:05:59.717 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:05:59.717 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:06:00.115 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:06:00.115 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:06:00.115 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:06:00.115 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:06:00.115 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:06:00.115 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:06:00.115 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:06:00.115 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:06:00.438 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-26T00:06:00.439 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:06:00.440 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:06:00.440 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:06:00.815 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-03-26T00:06:00.815 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-26T00:06:00.816 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-26T00:06:00.816 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-26T00:06:00.816 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-26T00:06:00.816 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-26T00:06:00.916 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-26T00:06:00.916 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:06:00.917 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:06:00.917 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:06:00.917 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:06:00.917 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:06:00.918 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:06:00.918 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:06:01.216 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-26T00:06:01.216 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:06:01.216 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:06:01.217 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:06:01.595 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-03-26T00:06:01.595 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-26T00:06:01.595 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-26T00:06:01.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-26T00:06:01.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-26T00:06:01.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-26T00:06:01.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-26T00:06:01.596 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-26T00:06:01.596 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-26T00:06:02.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-26T00:06:02.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-26T00:06:02.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-26T00:06:02.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-26T00:06:02.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-26T00:06:02.287 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-26T00:06:02.287 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:06:02.288 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:06:02.288 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:06:02.288 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:06:02.288 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:06:02.289 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:06:02.289 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:06:02.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-26T00:06:02.597 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:06:02.598 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:06:02.598 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:06:02.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-03-26T00:06:02.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-26T00:06:02.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-26T00:06:02.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-26T00:06:02.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-26T00:06:02.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-26T00:06:02.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-26T00:06:02.957 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-26T00:06:02.957 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-26T00:06:03.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-26T00:06:03.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-26T00:06:03.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-26T00:06:03.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-26T00:06:03.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-26T00:06:03.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-26T00:06:03.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:06:03.728 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:06:03.728 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:06:03.728 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:06:03.728 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:06:03.728 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:06:03.729 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:06:04.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-26T00:06:04.038 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:06:04.039 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:06:04.039 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:06:04.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-03-26T00:06:04.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-26T00:06:04.410 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-26T00:06:04.410 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-26T00:06:04.410 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-26T00:06:04.410 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-26T00:06:04.410 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-26T00:06:04.410 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-26T00:06:04.410 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-26T00:06:04.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-26T00:06:04.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-26T00:06:04.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 13 )) 2024-03-26T00:06:04.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-26T00:06:04.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-03-26T00:06:05.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-26T00:06:05.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:06:05.580 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:06:05.580 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:06:05.580 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:06:05.580 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:06:05.581 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:06:05.581 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:06:05.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:06:05.884 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:06:05.885 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:06:05.885 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:06:06.239 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:06:06.239 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:06:06.239 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:06:06.239 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-26T00:06:06.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ3 2024-03-26T00:06:06.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2024-03-26T00:06:06.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-26T00:06:06.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-26T00:06:06.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ3 2024-03-26T00:06:06.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-26T00:06:06.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-26T00:06:06.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-26T00:06:06.656 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-26T00:06:06.674 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-26T00:06:07.262 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-26T00:06:07.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-26T00:06:07.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-26T00:06:07.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:06:07.507 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 3 % 2 2024-03-26T00:06:07.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-03-26T00:06:07.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-26T00:06:07.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2024-03-26T00:06:07.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:06:07.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:06:07.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-26T00:06:07.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:06:07.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2024-03-26T00:06:07.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:06:07.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:06:07.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-26T00:06:07.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:06:07.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-26T00:06:07.510 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:06:07.510 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:06:07.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:06:07.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:06:07.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:06:07.822 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:06:07.822 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2024-03-26T00:06:07.822 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:06:07.822 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:06:07.822 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:06:07.823 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:06:07.823 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:06:07.823 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2024-03-26T00:06:09.790 INFO:tasks.workunit.client.0.smithi129.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2024-03-26T00:06:10.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-26T00:06:10.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:06:10.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:06:10.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-26T00:06:10.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:06:10.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:06:10.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:06:10.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:06:10.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:06:10.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:06:10.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:06:10.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:06:10.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:06:10.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:06:10.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:06:10.331 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:06:10.331 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:06:10.331 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:06:10.336 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:06:10.336 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:06:10.336 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:06:10.336 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:06:10.336 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:06:10.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:06:10.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:06:10.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:06:10.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:06:10.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:06:10.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:06:10.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:06:10.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:06:10.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:06:10.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:06:10.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-26T00:06:10.337 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.1 2024-03-26T00:06:10.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:06:10.338 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-26T00:06:10.338 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-26T00:06:10.338 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:06:10.338 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:06:10.338 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:06:10.367 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:06:10.366+0000 7f976a2fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:06:10.370 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:06:10.370+0000 7f976a2fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:06:10.372 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:06:10.370+0000 7f976a2fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:06:10.666 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-26T00:06:10.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:06:10.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:06:10.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:06:10.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:06:10.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:06:10.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:06:10.667 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:06:10.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:06:10.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:06:10.901 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:06:10.898+0000 7f976a2fb7c0 -1 Falling back to public interface 2024-03-26T00:06:10.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:06:11.990 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:06:11.990 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:06:11.990 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:06:11.990 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:06:11.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:06:11.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:06:12.300 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:06:13.302 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:06:13.302 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:06:13.302 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:06:13.302 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:06:13.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:06:13.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:06:13.457 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:06:13.454+0000 7f976a2fb7c0 -1 osd.1 54 log_to_monitors true 2024-03-26T00:06:13.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:06:14.718 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:06:14.718 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:06:14.718 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:06:14.719 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:06:14.719 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:06:14.719 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:06:15.044 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1 up in weight 1 up_from 59 up_thru 59 down_at 55 last_clean_interval [41,54) [v2:127.0.0.1:6810/1230384495,v1:127.0.0.1:6811/1230384495] [v2:127.0.0.1:6812/1230384495,v1:127.0.0.1:6813/1230384495] exists,up c51d3969-0ab9-40e5-91ca-849223a6c0c5 2024-03-26T00:06:15.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:06:15.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:06:15.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:06:15.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:06:15.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:06:15.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:06:15.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:06:15.046 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:06:15.046 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:06:15.046 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:06:15.046 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:06:15.046 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:06:15.046 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:06:15.159 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:06:15.159 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:06:15.159 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:06:15.159 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:06:15.159 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:06:15.159 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:06:15.554 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:06:15.554 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:06:15.554 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-26T00:06:15.555 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:06:15.555 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:06:15.555 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:06:15.745 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364807 2024-03-26T00:06:15.745 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364807 2024-03-26T00:06:15.745 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-214748364807' 2024-03-26T00:06:15.745 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:06:15.746 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:06:15.895 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070466 2024-03-26T00:06:15.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070466 2024-03-26T00:06:15.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-214748364807 1-253403070466' 2024-03-26T00:06:15.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:06:15.896 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:06:16.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509460 2024-03-26T00:06:16.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509460 2024-03-26T00:06:16.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-214748364807 1-253403070466 2-64424509460' 2024-03-26T00:06:16.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:06:16.038 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-214748364807 2024-03-26T00:06:16.038 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:06:16.039 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:06:16.040 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-214748364807 2024-03-26T00:06:16.040 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:06:16.041 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364807 2024-03-26T00:06:16.041 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 214748364807' 2024-03-26T00:06:16.041 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 214748364807 2024-03-26T00:06:16.042 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:06:16.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364807 -lt 214748364807 2024-03-26T00:06:16.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:06:16.374 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-253403070466 2024-03-26T00:06:16.375 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:06:16.376 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:06:16.377 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-253403070466 2024-03-26T00:06:16.377 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:06:16.378 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070466 2024-03-26T00:06:16.378 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 253403070466' 2024-03-26T00:06:16.378 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 253403070466 2024-03-26T00:06:16.379 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:06:16.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070466 -lt 253403070466 2024-03-26T00:06:16.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:06:16.713 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509460 2024-03-26T00:06:16.713 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:06:16.715 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:06:16.715 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509460 2024-03-26T00:06:16.716 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:06:16.717 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509460 2024-03-26T00:06:16.717 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-03-26T00:06:16.717 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 64424509460 2024-03-26T00:06:16.717 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:06:17.057 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509460 -lt 64424509460 2024-03-26T00:06:17.058 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:06:17.058 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:06:17.058 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:06:17.446 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:06:17.446 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:06:17.447 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:06:17.447 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:06:17.447 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:06:17.447 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:06:17.447 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:06:17.447 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:06:17.774 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:06:17.775 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:06:17.775 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:06:17.775 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:06:18.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:06:18.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:06:18.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:06:18.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-26T00:06:18.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ4 2024-03-26T00:06:18.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2024-03-26T00:06:18.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-26T00:06:18.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-26T00:06:18.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ4 2024-03-26T00:06:18.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-26T00:06:18.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-26T00:06:18.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-26T00:06:18.626 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-26T00:06:18.644 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-26T00:06:19.061 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-26T00:06:19.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-26T00:06:19.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-26T00:06:19.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:06:19.314 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 4 % 2 2024-03-26T00:06:19.315 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-03-26T00:06:19.316 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-26T00:06:19.316 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3638: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2024-03-26T00:06:19.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2024-03-26T00:06:19.804 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3642: corrupt_scrub_erasure: echo -n bad-val 2024-03-26T00:06:19.804 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3643: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-03-26T00:06:19.804 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:06:19.804 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:06:19.804 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-26T00:06:19.804 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:06:19.804 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-03-26T00:06:19.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:06:19.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:06:19.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-26T00:06:19.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:06:19.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-26T00:06:19.806 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:06:19.806 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:06:19.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:06:19.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:06:19.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:06:20.131 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:06:20.131 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-03-26T00:06:20.131 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:06:20.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:06:20.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:06:20.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:06:20.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:06:20.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-03-26T00:06:22.552 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-26T00:06:22.552 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:06:22.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:06:22.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-26T00:06:22.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:06:22.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:06:22.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:06:22.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:06:22.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:06:22.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:06:22.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-26T00:06:22.554 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-26T00:06:22.554 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:06:22.554 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:06:22.554 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:06:22.555 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:06:22.556 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:06:22.556 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:06:22.556 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:06:22.556 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:06:22.556 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:06:22.556 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:06:22.556 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:06:22.556 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:06:22.556 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:06:22.556 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:06:22.556 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:06:22.556 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:06:22.557 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:06:22.557 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:06:22.557 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:06:22.557 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:06:22.557 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-26T00:06:22.558 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-26T00:06:22.558 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.0 2024-03-26T00:06:22.558 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:06:22.558 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-26T00:06:22.559 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-26T00:06:22.559 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:06:22.559 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:06:22.559 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:06:22.590 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:06:22.590+0000 7f50492c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:06:22.593 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:06:22.590+0000 7f50492c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:06:22.594 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:06:22.594+0000 7f50492c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:06:22.878 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-26T00:06:22.879 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:06:22.879 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-26T00:06:22.879 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:06:22.879 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:06:22.879 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:06:22.879 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:06:22.879 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:06:22.879 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:06:22.879 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:06:23.129 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:06:23.126+0000 7f50492c57c0 -1 Falling back to public interface 2024-03-26T00:06:23.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:06:24.219 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:06:24.219 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:06:24.219 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:06:24.219 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:06:24.220 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:06:24.220 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:06:24.557 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:06:25.559 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:06:25.559 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:06:25.559 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:06:25.559 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:06:25.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:06:25.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:06:25.672 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:06:25.670+0000 7f50492c57c0 -1 osd.0 63 log_to_monitors true 2024-03-26T00:06:25.936 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:06:26.938 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:06:26.938 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:06:26.938 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:06:26.938 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:06:26.938 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:06:26.939 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:06:27.584 INFO:tasks.workunit.client.0.smithi129.stderr:osd.0 up in weight 1 up_from 68 up_thru 68 down_at 64 last_clean_interval [50,63) [v2:127.0.0.1:6802/715722512,v1:127.0.0.1:6803/715722512] [v2:127.0.0.1:6804/715722512,v1:127.0.0.1:6805/715722512] exists,up 0eae8a05-f177-416a-8c75-3129f39d10de 2024-03-26T00:06:27.584 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:06:27.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:06:27.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:06:27.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:06:27.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:06:27.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:06:27.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:06:27.586 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:06:27.586 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:06:27.586 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:06:27.586 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:06:27.586 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:06:27.586 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:06:27.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:06:27.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:06:27.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:06:27.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:06:27.705 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:06:27.705 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:06:28.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:06:28.113 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:06:28.113 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-26T00:06:28.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:06:28.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:06:28.113 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:06:28.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=292057776130 2024-03-26T00:06:28.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 292057776130 2024-03-26T00:06:28.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-292057776130' 2024-03-26T00:06:28.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:06:28.281 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:06:28.423 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070469 2024-03-26T00:06:28.423 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070469 2024-03-26T00:06:28.423 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-292057776130 1-253403070469' 2024-03-26T00:06:28.423 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:06:28.423 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:06:28.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509464 2024-03-26T00:06:28.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509464 2024-03-26T00:06:28.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-292057776130 1-253403070469 2-64424509464' 2024-03-26T00:06:28.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:06:28.569 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-292057776130 2024-03-26T00:06:28.569 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:06:28.569 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:06:28.570 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-292057776130 2024-03-26T00:06:28.570 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:06:28.571 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=292057776130 2024-03-26T00:06:28.571 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 292057776130' 2024-03-26T00:06:28.571 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 292057776130 2024-03-26T00:06:28.572 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:06:28.882 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 292057776130 2024-03-26T00:06:28.882 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:06:29.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:06:29.885 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:06:30.206 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 292057776130 2024-03-26T00:06:30.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:06:31.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-26T00:06:31.208 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:06:31.533 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 292057776130 -lt 292057776130 2024-03-26T00:06:31.534 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:06:31.535 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-253403070469 2024-03-26T00:06:31.535 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:06:31.537 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:06:31.538 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-253403070469 2024-03-26T00:06:31.538 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:06:31.539 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070469 2024-03-26T00:06:31.540 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 253403070469' 2024-03-26T00:06:31.540 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 253403070469 2024-03-26T00:06:31.540 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:06:31.856 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070470 -lt 253403070469 2024-03-26T00:06:31.856 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:06:31.857 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509464 2024-03-26T00:06:31.857 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:06:31.858 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:06:31.859 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509464 2024-03-26T00:06:31.859 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:06:31.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509464 2024-03-26T00:06:31.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-03-26T00:06:31.860 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 64424509464 2024-03-26T00:06:31.861 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:06:32.195 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509464 -lt 64424509464 2024-03-26T00:06:32.195 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:06:32.196 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:06:32.196 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:06:32.633 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:06:32.633 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:06:32.634 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:06:32.634 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:06:32.634 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:06:32.635 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:06:32.635 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:06:32.635 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:06:32.996 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:06:32.997 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:06:32.998 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:06:32.998 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:06:33.363 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:06:33.363 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:06:33.363 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:06:33.363 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3644: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-03-26T00:06:33.363 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:06:33.363 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:06:33.363 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-03-26T00:06:33.363 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:06:33.363 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-03-26T00:06:33.363 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:06:33.364 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:06:33.364 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-03-26T00:06:33.364 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:06:33.364 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-26T00:06:33.364 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:06:33.364 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:06:33.364 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:06:33.364 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:06:33.364 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:06:33.672 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:06:33.672 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-03-26T00:06:33.672 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:06:33.672 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:06:33.672 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-03-26T00:06:33.672 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:06:33.672 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-03-26T00:06:33.672 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2024-03-26T00:06:36.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-03-26T00:06:36.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:06:36.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:06:36.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-26T00:06:36.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:06:36.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-26T00:06:36.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:06:36.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:06:36.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:06:36.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:06:36.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-26T00:06:36.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-26T00:06:36.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:06:36.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:06:36.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:06:36.166 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:06:36.166 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:06:36.166 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:06:36.166 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:06:36.166 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:06:36.166 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:06:36.168 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:06:36.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:06:36.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:06:36.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:06:36.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:06:36.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:06:36.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:06:36.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:06:36.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:06:36.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:06:36.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:06:36.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-26T00:06:36.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-26T00:06:36.169 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.2 2024-03-26T00:06:36.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:06:36.169 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-26T00:06:36.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-26T00:06:36.170 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:06:36.170 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:06:36.170 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:06:36.201 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:06:36.198+0000 7ff101cf77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:06:36.204 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:06:36.202+0000 7ff101cf77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:06:36.206 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:06:36.206+0000 7ff101cf77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:06:36.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-26T00:06:36.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:06:36.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-26T00:06:36.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:06:36.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:06:36.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:06:36.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:06:36.500 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:06:36.500 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:06:36.500 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:06:36.745 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:06:36.742+0000 7ff101cf77c0 -1 Falling back to public interface 2024-03-26T00:06:36.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:06:37.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:06:37.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:06:37.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:06:37.832 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:06:37.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:06:37.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:06:38.156 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:06:39.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:06:39.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:06:39.158 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:06:39.158 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:06:39.158 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:06:39.158 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:06:39.470 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:06:39.474+0000 7ff101cf77c0 -1 osd.2 70 log_to_monitors true 2024-03-26T00:06:39.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:06:40.483 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:06:40.483 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:06:40.483 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:06:40.483 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:06:40.483 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:06:40.483 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:06:40.868 INFO:tasks.workunit.client.0.smithi129.stderr:osd.2 up in weight 1 up_from 75 up_thru 75 down_at 71 last_clean_interval [15,70) [v2:127.0.0.1:6818/1270436586,v1:127.0.0.1:6819/1270436586] [v2:127.0.0.1:6820/1270436586,v1:127.0.0.1:6821/1270436586] exists,up 27be98cf-bbe7-4a2e-99cb-ddbe940370a6 2024-03-26T00:06:40.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:06:40.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:06:40.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:06:40.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:06:40.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:06:40.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:06:40.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:06:40.869 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:06:40.870 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:06:40.870 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:06:40.871 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:06:40.871 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:06:40.871 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:06:40.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:06:40.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:06:40.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:06:40.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:06:40.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:06:40.988 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:06:41.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:06:41.392 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:06:41.392 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-26T00:06:41.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:06:41.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:06:41.392 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:06:41.558 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=292057776134 2024-03-26T00:06:41.558 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 292057776134 2024-03-26T00:06:41.558 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-292057776134' 2024-03-26T00:06:41.559 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:06:41.559 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:06:41.705 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070473 2024-03-26T00:06:41.705 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070473 2024-03-26T00:06:41.705 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-292057776134 1-253403070473' 2024-03-26T00:06:41.705 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:06:41.705 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:06:41.847 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547202 2024-03-26T00:06:41.848 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547202 2024-03-26T00:06:41.848 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-292057776134 1-253403070473 2-322122547202' 2024-03-26T00:06:41.848 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:06:41.848 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-292057776134 2024-03-26T00:06:41.849 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:06:41.850 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:06:41.850 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-292057776134 2024-03-26T00:06:41.850 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:06:41.851 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=292057776134 2024-03-26T00:06:41.851 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 292057776134' 2024-03-26T00:06:41.851 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 292057776134 2024-03-26T00:06:41.851 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:06:42.156 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 292057776132 -lt 292057776134 2024-03-26T00:06:42.156 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:06:43.158 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:06:43.159 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:06:43.475 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 292057776134 -lt 292057776134 2024-03-26T00:06:43.475 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:06:43.475 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-253403070473 2024-03-26T00:06:43.476 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:06:43.478 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:06:43.479 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-253403070473 2024-03-26T00:06:43.479 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:06:43.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070473 2024-03-26T00:06:43.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 253403070473' 2024-03-26T00:06:43.481 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 253403070473 2024-03-26T00:06:43.482 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:06:43.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070473 -lt 253403070473 2024-03-26T00:06:43.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:06:43.806 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-322122547202 2024-03-26T00:06:43.806 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:06:43.808 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:06:43.809 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-322122547202 2024-03-26T00:06:43.809 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:06:43.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547202 2024-03-26T00:06:43.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 322122547202' 2024-03-26T00:06:43.810 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 322122547202 2024-03-26T00:06:43.810 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:06:44.146 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547202 -lt 322122547202 2024-03-26T00:06:44.146 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:06:44.147 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:06:44.147 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:06:44.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:06:44.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:06:44.568 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:06:44.568 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:06:44.568 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:06:44.568 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:06:44.568 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:06:44.568 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:06:44.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:06:44.920 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:06:44.921 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:06:44.921 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:06:45.277 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:06:45.277 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:06:45.277 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:06:45.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3645: corrupt_scrub_erasure: echo -n val3-EOBJ4 2024-03-26T00:06:45.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3646: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-03-26T00:06:45.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:06:45.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:06:45.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-03-26T00:06:45.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:06:45.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-03-26T00:06:45.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:06:45.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:06:45.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-03-26T00:06:45.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:06:45.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-26T00:06:45.278 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:06:45.279 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:06:45.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:06:45.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:06:45.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:06:45.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:06:45.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-03-26T00:06:45.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:06:45.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:06:45.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-03-26T00:06:45.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:06:45.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-03-26T00:06:45.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-03-26T00:06:47.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-03-26T00:06:47.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:06:47.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:06:47.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-26T00:06:47.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:06:47.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-26T00:06:47.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:06:47.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:06:47.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:06:47.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:06:47.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-26T00:06:47.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-26T00:06:47.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:06:47.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:06:47.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:06:47.950 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:06:47.950 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:06:47.950 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:06:47.950 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:06:47.950 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:06:47.950 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:06:47.950 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:06:47.951 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:06:47.951 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:06:47.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:06:47.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:06:47.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:06:47.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:06:47.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:06:47.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:06:47.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:06:47.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:06:47.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-26T00:06:47.953 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-26T00:06:47.953 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.2 2024-03-26T00:06:47.953 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:06:47.953 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-26T00:06:47.954 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-26T00:06:47.954 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:06:47.955 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:06:47.955 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:06:47.986 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:06:47.989+0000 7ff35f3c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:06:47.989 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:06:47.989+0000 7ff35f3c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:06:47.990 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:06:47.993+0000 7ff35f3c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:06:48.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-26T00:06:48.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:06:48.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-26T00:06:48.268 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:06:48.268 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:06:48.268 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:06:48.268 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:06:48.268 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:06:48.268 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:06:48.268 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:06:48.536 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:06:48.537+0000 7ff35f3c77c0 -1 Falling back to public interface 2024-03-26T00:06:48.601 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:06:49.603 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:06:49.603 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:06:49.603 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:06:49.603 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:06:49.604 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:06:49.604 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:06:49.905 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:06:50.906 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:06:50.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:06:50.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:06:50.907 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:06:50.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:06:50.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:06:51.218 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:06:51.217+0000 7ff35f3c77c0 -1 osd.2 77 log_to_monitors true 2024-03-26T00:06:51.233 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:06:52.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:06:52.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:06:52.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:06:52.236 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:06:52.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:06:52.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:06:52.561 INFO:tasks.workunit.client.0.smithi129.stderr:osd.2 up in weight 1 up_from 82 up_thru 82 down_at 78 last_clean_interval [75,77) [v2:127.0.0.1:6818/2888120274,v1:127.0.0.1:6819/2888120274] [v2:127.0.0.1:6820/2888120274,v1:127.0.0.1:6821/2888120274] exists,up 27be98cf-bbe7-4a2e-99cb-ddbe940370a6 2024-03-26T00:06:52.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:06:52.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:06:52.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:06:52.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:06:52.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:06:52.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:06:52.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:06:52.562 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:06:52.562 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:06:52.562 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:06:52.562 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:06:52.563 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:06:52.563 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:06:52.677 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:06:52.678 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:06:52.678 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:06:52.678 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:06:52.678 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:06:52.678 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:06:53.117 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:06:53.118 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:06:53.118 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-26T00:06:53.118 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:06:53.118 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:06:53.118 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:06:53.268 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=292057776137 2024-03-26T00:06:53.268 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 292057776137 2024-03-26T00:06:53.269 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-292057776137' 2024-03-26T00:06:53.269 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:06:53.269 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:06:53.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070476 2024-03-26T00:06:53.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070476 2024-03-26T00:06:53.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-292057776137 1-253403070476' 2024-03-26T00:06:53.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:06:53.413 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:06:53.556 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318274 2024-03-26T00:06:53.556 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318274 2024-03-26T00:06:53.556 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-292057776137 1-253403070476 2-352187318274' 2024-03-26T00:06:53.556 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:06:53.556 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-292057776137 2024-03-26T00:06:53.556 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:06:53.557 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:06:53.557 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-292057776137 2024-03-26T00:06:53.558 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:06:53.558 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=292057776137 2024-03-26T00:06:53.559 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 292057776137' 2024-03-26T00:06:53.559 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 292057776137 2024-03-26T00:06:53.559 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:06:53.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 292057776136 -lt 292057776137 2024-03-26T00:06:53.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:06:54.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:06:54.870 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:06:55.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 292057776137 -lt 292057776137 2024-03-26T00:06:55.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:06:55.180 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-253403070476 2024-03-26T00:06:55.180 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:06:55.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:06:55.183 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-253403070476 2024-03-26T00:06:55.183 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:06:55.185 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070476 2024-03-26T00:06:55.185 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 253403070476' 2024-03-26T00:06:55.185 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 253403070476 2024-03-26T00:06:55.185 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:06:55.525 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070477 -lt 253403070476 2024-03-26T00:06:55.525 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:06:55.526 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-352187318274 2024-03-26T00:06:55.526 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:06:55.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:06:55.528 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-352187318274 2024-03-26T00:06:55.528 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:06:55.530 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318274 2024-03-26T00:06:55.530 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 352187318274' 2024-03-26T00:06:55.530 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 352187318274 2024-03-26T00:06:55.531 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:06:55.867 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318274 -lt 352187318274 2024-03-26T00:06:55.867 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:06:55.868 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:06:55.868 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:06:56.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:06:56.258 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:06:56.259 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:06:56.260 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:06:56.260 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:06:56.260 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:06:56.260 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:06:56.260 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:06:56.573 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:06:56.574 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:06:56.574 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:06:56.574 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:06:56.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:06:56.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:06:56.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:06:56.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3647: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-03-26T00:06:56.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-26T00:06:56.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ5 2024-03-26T00:06:56.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2024-03-26T00:06:56.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-26T00:06:56.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-26T00:06:56.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ5 2024-03-26T00:06:56.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-26T00:06:56.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-26T00:06:56.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-26T00:06:57.325 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-26T00:06:57.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-26T00:06:57.844 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-26T00:06:57.862 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-26T00:06:57.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-26T00:06:57.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:06:58.204 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 5 % 2 2024-03-26T00:06:58.206 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-03-26T00:06:58.206 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-26T00:06:58.206 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3652: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2024-03-26T00:06:58.208 INFO:tasks.workunit.client.0.smithi129.stderr:2+0 records in 2024-03-26T00:06:58.208 INFO:tasks.workunit.client.0.smithi129.stderr:2+0 records out 2024-03-26T00:06:58.208 INFO:tasks.workunit.client.0.smithi129.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000168912 s, 24.2 MB/s 2024-03-26T00:06:58.208 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3653: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-26T00:06:58.208 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:06:58.208 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:06:58.208 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-26T00:06:58.209 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:06:58.209 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-26T00:06:58.209 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:06:58.209 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:06:58.209 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-26T00:06:58.209 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:06:58.209 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-26T00:06:58.210 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:06:58.210 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:06:58.210 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:06:58.210 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:06:58.210 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:06:59.522 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:06:59.523 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-26T00:06:59.523 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:06:59.523 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:06:59.523 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:06:59.523 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:06:59.558 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:06:59.558 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-26T00:07:01.574 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-26T00:07:01.574 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:07:01.574 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:07:01.574 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-26T00:07:01.575 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:07:01.575 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:07:01.575 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:07:01.575 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:07:01.575 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:07:01.575 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:07:01.575 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:07:01.575 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:07:01.575 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:07:01.575 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:07:01.575 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:07:01.575 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:07:01.575 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:07:01.576 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:07:01.576 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:07:01.576 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:07:01.576 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:07:01.577 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:07:01.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:07:01.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:07:01.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:07:01.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:07:01.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:07:01.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:07:01.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:07:01.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:07:01.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:07:01.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:07:01.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:07:01.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-26T00:07:01.578 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.1 2024-03-26T00:07:01.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:07:01.578 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-26T00:07:01.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-26T00:07:01.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:07:01.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:07:01.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:07:01.611 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:07:01.613+0000 7fc52d61b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:07:01.614 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:07:01.613+0000 7fc52d61b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:07:01.616 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:07:01.617+0000 7fc52d61b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:07:01.903 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-26T00:07:01.903 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:07:01.903 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:07:01.903 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:07:01.903 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:07:01.903 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:07:01.903 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:07:01.904 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:07:01.904 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:07:01.904 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:07:02.157 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:07:02.157+0000 7fc52d61b7c0 -1 Falling back to public interface 2024-03-26T00:07:02.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:07:03.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:07:03.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:07:03.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:07:03.230 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:07:03.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:07:03.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:07:03.548 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:07:04.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:07:04.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:07:04.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:07:04.550 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:07:04.551 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:07:04.551 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:07:04.871 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:07:04.884 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:07:04.885+0000 7fc52d61b7c0 -1 osd.1 86 log_to_monitors true 2024-03-26T00:07:05.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:07:05.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:07:05.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:07:05.873 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:07:05.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:07:05.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:07:06.381 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1 up in weight 1 up_from 91 up_thru 91 down_at 87 last_clean_interval [59,86) [v2:127.0.0.1:6810/3192125202,v1:127.0.0.1:6811/3192125202] [v2:127.0.0.1:6812/3192125202,v1:127.0.0.1:6813/3192125202] exists,up c51d3969-0ab9-40e5-91ca-849223a6c0c5 2024-03-26T00:07:06.381 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:07:06.381 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:07:06.381 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:07:06.381 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:07:06.381 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:07:06.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:07:06.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:07:06.382 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:07:06.382 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:07:06.382 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:07:06.382 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:07:06.383 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:07:06.383 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:07:06.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:07:06.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:07:06.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:07:06.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:07:06.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:07:06.497 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:07:06.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:07:06.914 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:07:06.914 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-26T00:07:06.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:07:06.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:07:06.915 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:07:07.064 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=292057776141 2024-03-26T00:07:07.064 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 292057776141 2024-03-26T00:07:07.064 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-292057776141' 2024-03-26T00:07:07.064 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:07:07.065 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:07:07.212 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=390842023938 2024-03-26T00:07:07.213 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 390842023938 2024-03-26T00:07:07.213 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-292057776141 1-390842023938' 2024-03-26T00:07:07.213 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:07:07.213 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:07:07.351 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318278 2024-03-26T00:07:07.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318278 2024-03-26T00:07:07.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-292057776141 1-390842023938 2-352187318278' 2024-03-26T00:07:07.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:07:07.352 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:07:07.352 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-292057776141 2024-03-26T00:07:07.353 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:07:07.354 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-292057776141 2024-03-26T00:07:07.354 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:07:07.355 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=292057776141 2024-03-26T00:07:07.355 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 292057776141' 2024-03-26T00:07:07.355 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 292057776141 2024-03-26T00:07:07.355 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:07:07.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 292057776140 -lt 292057776141 2024-03-26T00:07:07.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:07:08.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:07:08.650 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:07:09.011 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 292057776141 -lt 292057776141 2024-03-26T00:07:09.011 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:07:09.012 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-390842023938 2024-03-26T00:07:09.012 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:07:09.014 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:07:09.015 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-390842023938 2024-03-26T00:07:09.015 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:07:09.016 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=390842023938 2024-03-26T00:07:09.016 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 390842023938' 2024-03-26T00:07:09.017 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 390842023938 2024-03-26T00:07:09.017 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:07:09.356 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 390842023938 -lt 390842023938 2024-03-26T00:07:09.357 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:07:09.358 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-352187318278 2024-03-26T00:07:09.358 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:07:09.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:07:09.360 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-352187318278 2024-03-26T00:07:09.360 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:07:09.362 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318278 2024-03-26T00:07:09.362 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 352187318278' 2024-03-26T00:07:09.362 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 352187318278 2024-03-26T00:07:09.362 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:07:09.705 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318278 -lt 352187318278 2024-03-26T00:07:09.706 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:07:09.710 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:07:09.710 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:07:10.096 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:07:10.096 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:07:10.097 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:07:10.097 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:07:10.097 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:07:10.097 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:07:10.097 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:07:10.097 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:07:10.440 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-26T00:07:10.440 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:07:10.441 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:07:10.441 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:07:10.818 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-03-26T00:07:10.818 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-26T00:07:10.818 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-26T00:07:10.818 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-26T00:07:10.818 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-26T00:07:10.818 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-26T00:07:10.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-26T00:07:10.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:07:10.920 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:07:10.920 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:07:10.920 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:07:10.920 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:07:10.920 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:07:10.921 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:07:11.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-26T00:07:11.233 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:07:11.233 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:07:11.233 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:07:11.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-03-26T00:07:11.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-26T00:07:11.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-26T00:07:11.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-26T00:07:11.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-26T00:07:11.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-26T00:07:11.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-26T00:07:11.587 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-26T00:07:11.587 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-26T00:07:11.971 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-26T00:07:11.971 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-26T00:07:11.971 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-26T00:07:11.971 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-26T00:07:11.971 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-26T00:07:12.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-26T00:07:12.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:07:12.172 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:07:12.172 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:07:12.172 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:07:12.173 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:07:12.173 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:07:12.173 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:07:12.503 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:07:12.503 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:07:12.504 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:07:12.504 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:07:12.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:07:12.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:07:12.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:07:12.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-26T00:07:12.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ6 2024-03-26T00:07:12.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2024-03-26T00:07:12.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-26T00:07:12.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-26T00:07:12.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ6 2024-03-26T00:07:12.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-26T00:07:12.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-26T00:07:12.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-26T00:07:13.346 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-26T00:07:13.366 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-26T00:07:13.923 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-26T00:07:13.942 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-26T00:07:13.942 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-26T00:07:13.943 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:07:14.162 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 6 % 2 2024-03-26T00:07:14.164 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-03-26T00:07:14.164 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-26T00:07:14.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3657: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-03-26T00:07:14.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:07:14.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:07:14.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-26T00:07:14.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:07:14.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-03-26T00:07:14.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:07:14.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:07:14.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-26T00:07:14.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:07:14.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-26T00:07:14.165 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:07:14.165 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:07:14.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:07:14.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:07:14.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:07:14.477 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:07:14.477 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-03-26T00:07:14.477 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:07:14.477 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:07:14.477 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:07:14.477 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:07:14.478 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:07:14.478 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2024-03-26T00:07:16.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-26T00:07:16.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:07:16.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:07:16.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-26T00:07:16.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:07:16.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:07:16.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:07:16.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:07:16.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:07:16.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:07:16.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-26T00:07:16.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-26T00:07:16.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:07:16.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:07:16.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:07:16.908 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:07:16.908 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:07:16.908 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:07:16.908 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:07:16.908 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:07:16.908 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:07:16.908 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:07:16.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:07:16.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:07:16.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:07:16.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:07:16.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:07:16.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:07:16.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:07:16.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:07:16.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:07:16.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:07:16.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-26T00:07:16.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-26T00:07:16.911 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.0 2024-03-26T00:07:16.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:07:16.911 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-26T00:07:16.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-26T00:07:16.912 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:07:16.912 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:07:16.912 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:07:16.943 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:07:16.945+0000 7f1cd42ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:07:16.946 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:07:16.945+0000 7f1cd42ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:07:16.947 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:07:16.949+0000 7f1cd42ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:07:17.234 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-26T00:07:17.234 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:07:17.234 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-26T00:07:17.234 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:07:17.234 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:07:17.234 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:07:17.234 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:07:17.234 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:07:17.234 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:07:17.235 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:07:17.473 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:07:17.473+0000 7f1cd42ea7c0 -1 Falling back to public interface 2024-03-26T00:07:17.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:07:18.602 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:07:18.602 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:07:18.602 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:07:18.602 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:07:18.602 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:07:18.602 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:07:18.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:07:19.924 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:07:19.924 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:07:19.924 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:07:19.924 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:07:19.924 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:07:19.925 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:07:20.156 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:07:20.157+0000 7f1cd42ea7c0 -1 osd.0 95 log_to_monitors true 2024-03-26T00:07:20.295 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:07:21.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:07:21.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:07:21.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:07:21.298 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:07:21.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:07:21.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:07:21.797 INFO:tasks.workunit.client.0.smithi129.stderr:osd.0 up in weight 1 up_from 100 up_thru 100 down_at 96 last_clean_interval [68,95) [v2:127.0.0.1:6802/441925162,v1:127.0.0.1:6803/441925162] [v2:127.0.0.1:6804/441925162,v1:127.0.0.1:6805/441925162] exists,up 0eae8a05-f177-416a-8c75-3129f39d10de 2024-03-26T00:07:21.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:07:21.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:07:21.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:07:21.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:07:21.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:07:21.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:07:21.797 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:07:21.797 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:07:21.798 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:07:21.798 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:07:21.798 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:07:21.798 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:07:21.798 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:07:21.913 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:07:21.913 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:07:21.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:07:21.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:07:21.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:07:21.914 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:07:22.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:07:22.310 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:07:22.310 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-26T00:07:22.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:07:22.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:07:22.310 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:07:22.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=429496729602 2024-03-26T00:07:22.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 429496729602 2024-03-26T00:07:22.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-429496729602' 2024-03-26T00:07:22.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:07:22.515 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:07:22.656 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=390842023942 2024-03-26T00:07:22.718 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 390842023942 2024-03-26T00:07:22.718 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-429496729602 1-390842023942' 2024-03-26T00:07:22.718 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:07:22.719 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:07:22.799 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318282 2024-03-26T00:07:22.799 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318282 2024-03-26T00:07:22.799 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-429496729602 1-390842023942 2-352187318282' 2024-03-26T00:07:22.799 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:07:22.800 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-429496729602 2024-03-26T00:07:22.800 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:07:22.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:07:22.801 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-429496729602 2024-03-26T00:07:22.801 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:07:22.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=429496729602 2024-03-26T00:07:22.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 429496729602' 2024-03-26T00:07:22.802 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 429496729602 2024-03-26T00:07:22.803 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:07:23.103 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 429496729602 2024-03-26T00:07:23.103 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:07:24.105 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:07:24.106 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:07:24.422 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 429496729602 -lt 429496729602 2024-03-26T00:07:24.423 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:07:24.424 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-390842023942 2024-03-26T00:07:24.424 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:07:24.426 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:07:24.427 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-390842023942 2024-03-26T00:07:24.427 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:07:24.428 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=390842023942 2024-03-26T00:07:24.428 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 390842023942' 2024-03-26T00:07:24.428 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 390842023942 2024-03-26T00:07:24.429 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:07:24.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 390842023942 -lt 390842023942 2024-03-26T00:07:24.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:07:24.756 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-352187318282 2024-03-26T00:07:24.756 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:07:24.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:07:24.758 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-352187318282 2024-03-26T00:07:24.758 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:07:24.759 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318282 2024-03-26T00:07:24.759 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 352187318282' 2024-03-26T00:07:24.759 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 352187318282 2024-03-26T00:07:24.760 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:07:25.074 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318282 -lt 352187318282 2024-03-26T00:07:25.075 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:07:25.076 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:07:25.076 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:07:25.444 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:07:25.444 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:07:25.445 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:07:25.445 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:07:25.445 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:07:25.445 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:07:25.445 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:07:25.445 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:07:25.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:07:25.752 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:07:25.753 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:07:25.753 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:07:26.118 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:07:26.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:07:26.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:07:26.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3658: corrupt_scrub_erasure: echo -n bad-val 2024-03-26T00:07:26.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3659: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-03-26T00:07:26.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:07:26.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:07:26.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-26T00:07:26.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:07:26.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-03-26T00:07:26.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:07:26.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:07:26.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-26T00:07:26.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:07:26.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-26T00:07:26.120 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:07:26.120 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:07:26.120 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:07:26.120 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:07:26.120 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:07:26.428 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:07:26.429 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-03-26T00:07:26.429 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:07:26.429 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:07:26.429 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:07:26.429 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:07:26.429 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:07:26.429 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-03-26T00:07:28.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-26T00:07:28.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:07:28.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:07:28.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-26T00:07:28.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:07:28.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:07:28.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:07:28.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:07:28.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:07:28.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:07:28.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:07:28.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:07:28.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:07:28.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:07:28.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:07:28.689 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:07:28.689 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:07:28.689 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:07:28.691 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:07:28.691 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:07:28.691 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:07:28.691 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:07:28.691 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:07:28.691 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:07:28.691 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:07:28.691 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:07:28.691 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:07:28.692 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:07:28.692 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:07:28.692 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:07:28.692 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:07:28.692 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:07:28.692 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:07:28.692 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-26T00:07:28.692 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.1 2024-03-26T00:07:28.693 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:07:28.693 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-26T00:07:28.693 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-26T00:07:28.693 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:07:28.693 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:07:28.694 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:07:28.724 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:07:28.725+0000 7f4cb59487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:07:28.727 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:07:28.729+0000 7f4cb59487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:07:28.729 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:07:28.729+0000 7f4cb59487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:07:29.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-26T00:07:29.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:07:29.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:07:29.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:07:29.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:07:29.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:07:29.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:07:29.070 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:07:29.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:07:29.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:07:29.256 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:07:29.257+0000 7f4cb59487c0 -1 Falling back to public interface 2024-03-26T00:07:29.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:07:30.408 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:07:30.408 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:07:30.408 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:07:30.408 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:07:30.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:07:30.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:07:30.717 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:07:31.719 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:07:31.719 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:07:31.719 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:07:31.719 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:07:31.720 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:07:31.720 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:07:31.892 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:07:31.893+0000 7f4cb59487c0 -1 osd.1 102 log_to_monitors true 2024-03-26T00:07:32.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:07:33.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:07:33.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:07:33.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:07:33.048 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:07:33.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:07:33.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:07:33.390 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1 up in weight 1 up_from 107 up_thru 107 down_at 103 last_clean_interval [91,102) [v2:127.0.0.1:6810/854001000,v1:127.0.0.1:6811/854001000] [v2:127.0.0.1:6812/854001000,v1:127.0.0.1:6813/854001000] exists,up c51d3969-0ab9-40e5-91ca-849223a6c0c5 2024-03-26T00:07:33.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:07:33.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:07:33.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:07:33.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:07:33.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:07:33.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:07:33.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:07:33.391 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:07:33.391 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:07:33.391 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:07:33.392 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:07:33.392 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:07:33.392 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:07:33.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:07:33.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:07:33.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:07:33.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:07:33.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:07:33.496 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:07:33.815 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:07:33.816 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:07:33.816 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-26T00:07:33.816 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:07:33.816 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:07:33.816 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:07:34.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=429496729605 2024-03-26T00:07:34.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 429496729605 2024-03-26T00:07:34.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-429496729605' 2024-03-26T00:07:34.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:07:34.042 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:07:34.224 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=459561500674 2024-03-26T00:07:34.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 459561500674 2024-03-26T00:07:34.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-429496729605 1-459561500674' 2024-03-26T00:07:34.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:07:34.225 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:07:34.369 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318285 2024-03-26T00:07:34.369 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318285 2024-03-26T00:07:34.369 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-429496729605 1-459561500674 2-352187318285' 2024-03-26T00:07:34.369 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:07:34.369 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-429496729605 2024-03-26T00:07:34.370 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:07:34.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:07:34.371 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-429496729605 2024-03-26T00:07:34.371 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:07:34.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=429496729605 2024-03-26T00:07:34.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 429496729605' 2024-03-26T00:07:34.373 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 429496729605 2024-03-26T00:07:34.373 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:07:34.672 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 429496729605 -lt 429496729605 2024-03-26T00:07:34.672 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:07:34.673 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-459561500674 2024-03-26T00:07:34.673 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:07:34.678 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:07:34.678 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-459561500674 2024-03-26T00:07:34.678 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:07:34.678 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=459561500674 2024-03-26T00:07:34.678 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 459561500674' 2024-03-26T00:07:34.678 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 459561500674 2024-03-26T00:07:34.678 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:07:35.030 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 459561500674 -lt 459561500674 2024-03-26T00:07:35.030 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:07:35.031 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-352187318285 2024-03-26T00:07:35.031 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:07:35.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:07:35.033 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-352187318285 2024-03-26T00:07:35.034 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:07:35.035 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318285 2024-03-26T00:07:35.035 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 352187318285' 2024-03-26T00:07:35.035 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 352187318285 2024-03-26T00:07:35.035 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:07:35.356 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318284 -lt 352187318285 2024-03-26T00:07:35.356 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:07:36.358 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:07:36.359 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:07:36.679 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318285 -lt 352187318285 2024-03-26T00:07:36.680 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:07:36.680 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:07:36.680 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:07:37.079 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:07:37.079 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:07:37.079 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:07:37.079 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:07:37.079 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:07:37.079 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:07:37.080 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:07:37.080 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:07:37.417 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:07:37.418 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:07:37.418 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:07:37.418 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:07:37.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:07:37.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:07:37.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:07:37.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-26T00:07:37.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ7 2024-03-26T00:07:37.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2024-03-26T00:07:37.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-26T00:07:37.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-26T00:07:37.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ7 2024-03-26T00:07:37.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-26T00:07:37.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-26T00:07:37.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-26T00:07:38.176 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-26T00:07:38.195 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-26T00:07:38.760 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-26T00:07:38.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-26T00:07:38.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-26T00:07:38.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:07:39.084 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 7 % 2 2024-03-26T00:07:39.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-03-26T00:07:39.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-26T00:07:39.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3663: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-03-26T00:07:39.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3664: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-03-26T00:07:39.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3665: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2024-03-26T00:07:39.318 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3668: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-03-26T00:07:39.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:07:39.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:07:39.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-26T00:07:39.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:07:39.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-03-26T00:07:39.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:07:39.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:07:39.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-26T00:07:39.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:07:39.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-26T00:07:39.319 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:07:39.319 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:07:39.320 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:07:39.320 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:07:39.320 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:07:39.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:07:39.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-03-26T00:07:39.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:07:39.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:07:39.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:07:39.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:07:39.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:07:39.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2024-03-26T00:07:42.001 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-26T00:07:42.001 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:07:42.001 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:07:42.001 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-26T00:07:42.001 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:07:42.001 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:07:42.001 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:07:42.002 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:07:42.002 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:07:42.002 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:07:42.002 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-26T00:07:42.002 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-26T00:07:42.002 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:07:42.002 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:07:42.002 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:07:42.002 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:07:42.002 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:07:42.002 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:07:42.003 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:07:42.003 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:07:42.004 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:07:42.004 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:07:42.004 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:07:42.004 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:07:42.004 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:07:42.005 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:07:42.005 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:07:42.005 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:07:42.005 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:07:42.005 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:07:42.005 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:07:42.005 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:07:42.005 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-26T00:07:42.006 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-26T00:07:42.006 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.0 2024-03-26T00:07:42.006 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:07:42.006 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-26T00:07:42.006 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-26T00:07:42.007 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:07:42.007 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:07:42.007 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:07:42.039 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:07:42.040+0000 7feb56b567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:07:42.042 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:07:42.040+0000 7feb56b567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:07:42.044 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:07:42.044+0000 7feb56b567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:07:42.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-26T00:07:42.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:07:42.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-26T00:07:42.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:07:42.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:07:42.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:07:42.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:07:42.342 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:07:42.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:07:42.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:07:42.577 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:07:42.576+0000 7feb56b567c0 -1 Falling back to public interface 2024-03-26T00:07:42.673 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:07:43.675 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:07:43.675 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:07:43.675 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:07:43.675 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:07:43.675 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:07:43.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:07:43.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:07:44.993 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:07:44.993 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:07:44.993 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:07:44.994 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:07:44.994 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:07:44.994 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:07:45.239 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:07:45.240+0000 7feb56b567c0 -1 osd.0 111 log_to_monitors true 2024-03-26T00:07:45.301 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:07:46.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:07:46.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:07:46.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:07:46.303 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:07:46.304 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:07:46.304 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:07:46.747 INFO:tasks.workunit.client.0.smithi129.stderr:osd.0 up in weight 1 up_from 116 up_thru 116 down_at 112 last_clean_interval [100,111) [v2:127.0.0.1:6802/3930531967,v1:127.0.0.1:6803/3930531967] [v2:127.0.0.1:6804/3930531967,v1:127.0.0.1:6805/3930531967] exists,up 0eae8a05-f177-416a-8c75-3129f39d10de 2024-03-26T00:07:46.747 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:07:46.747 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:07:46.747 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:07:46.747 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:07:46.747 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:07:46.748 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:07:46.748 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:07:46.748 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:07:46.749 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:07:46.749 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:07:46.750 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:07:46.750 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:07:46.750 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:07:46.855 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:07:46.856 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:07:46.856 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:07:46.856 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:07:46.856 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:07:46.856 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:07:47.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:07:47.281 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:07:47.281 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-26T00:07:47.282 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:07:47.282 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:07:47.282 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:07:47.436 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=498216206338 2024-03-26T00:07:47.436 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 498216206338 2024-03-26T00:07:47.436 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-498216206338' 2024-03-26T00:07:47.436 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:07:47.436 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:07:47.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=459561500678 2024-03-26T00:07:47.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 459561500678 2024-03-26T00:07:47.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-498216206338 1-459561500678' 2024-03-26T00:07:47.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:07:47.577 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:07:47.719 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318289 2024-03-26T00:07:47.719 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318289 2024-03-26T00:07:47.719 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-498216206338 1-459561500678 2-352187318289' 2024-03-26T00:07:47.719 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:07:47.720 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-498216206338 2024-03-26T00:07:47.720 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:07:47.720 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:07:47.721 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-498216206338 2024-03-26T00:07:47.721 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:07:47.723 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=498216206338 2024-03-26T00:07:47.723 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 498216206338' 2024-03-26T00:07:47.723 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 498216206338 2024-03-26T00:07:47.723 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:07:48.019 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 498216206338 2024-03-26T00:07:48.020 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:07:49.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:07:49.022 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:07:49.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 498216206338 -lt 498216206338 2024-03-26T00:07:49.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:07:49.326 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-459561500678 2024-03-26T00:07:49.327 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:07:49.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:07:49.330 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-459561500678 2024-03-26T00:07:49.330 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:07:49.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=459561500678 2024-03-26T00:07:49.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 459561500678' 2024-03-26T00:07:49.331 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 459561500678 2024-03-26T00:07:49.332 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:07:49.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 459561500678 -lt 459561500678 2024-03-26T00:07:49.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:07:49.671 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-352187318289 2024-03-26T00:07:49.671 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:07:49.672 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:07:49.674 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-352187318289 2024-03-26T00:07:49.674 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:07:49.675 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318289 2024-03-26T00:07:49.675 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 352187318289' 2024-03-26T00:07:49.675 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 352187318289 2024-03-26T00:07:49.675 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:07:50.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318289 -lt 352187318289 2024-03-26T00:07:50.046 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:07:50.046 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:07:50.047 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:07:50.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:07:50.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:07:50.390 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:07:50.390 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:07:50.390 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:07:50.390 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:07:50.390 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:07:50.390 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:07:50.687 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:07:50.687 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:07:50.688 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:07:50.688 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:07:51.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:07:51.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:07:51.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:07:51.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3669: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-03-26T00:07:51.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:07:51.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:07:51.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-26T00:07:51.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:07:51.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-03-26T00:07:51.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:07:51.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:07:51.063 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-26T00:07:51.063 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:07:51.063 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-26T00:07:51.063 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:07:51.063 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:07:51.063 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:07:51.063 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:07:51.063 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:07:51.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:07:51.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-03-26T00:07:51.372 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:07:51.372 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:07:51.372 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:07:51.372 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:07:51.372 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:07:51.372 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-03-26T00:07:53.912 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-26T00:07:53.912 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:07:53.912 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:07:53.912 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-26T00:07:53.912 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:07:53.912 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:07:53.913 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:07:53.913 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:07:53.913 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:07:53.913 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:07:53.913 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-26T00:07:53.913 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-26T00:07:53.913 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:07:53.913 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:07:53.913 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:07:53.913 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:07:53.913 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:07:53.913 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:07:53.914 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:07:53.914 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:07:53.914 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:07:53.915 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:07:53.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:07:53.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:07:53.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:07:53.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:07:53.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:07:53.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:07:53.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:07:53.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:07:53.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:07:53.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:07:53.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-26T00:07:53.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-26T00:07:53.920 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.0 2024-03-26T00:07:53.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:07:53.920 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-26T00:07:53.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-26T00:07:53.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:07:53.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:07:53.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:07:53.954 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:07:53.952+0000 7f02916f77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:07:53.956 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:07:53.956+0000 7f02916f77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:07:53.958 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:07:53.960+0000 7f02916f77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:07:54.245 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-26T00:07:54.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:07:54.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-26T00:07:54.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:07:54.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:07:54.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:07:54.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:07:54.246 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:07:54.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:07:54.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:07:54.493 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:07:54.492+0000 7f02916f77c0 -1 Falling back to public interface 2024-03-26T00:07:54.576 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:07:55.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:07:55.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:07:55.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:07:55.578 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:07:55.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:07:55.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:07:55.883 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:07:56.885 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:07:56.885 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:07:56.885 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:07:56.886 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:07:56.886 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:07:56.886 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:07:57.222 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:07:57.224+0000 7f02916f77c0 -1 osd.0 118 log_to_monitors true 2024-03-26T00:07:57.258 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:07:58.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:07:58.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:07:58.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:07:58.260 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:07:58.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:07:58.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:07:58.613 INFO:tasks.workunit.client.0.smithi129.stderr:osd.0 up in weight 1 up_from 122 up_thru 122 down_at 119 last_clean_interval [116,118) [v2:127.0.0.1:6802/4019006383,v1:127.0.0.1:6803/4019006383] [v2:127.0.0.1:6804/4019006383,v1:127.0.0.1:6805/4019006383] exists,up 0eae8a05-f177-416a-8c75-3129f39d10de 2024-03-26T00:07:58.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:07:58.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:07:58.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:07:58.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:07:58.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:07:58.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:07:58.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:07:58.614 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:07:58.614 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:07:58.615 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:07:58.615 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:07:58.615 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:07:58.615 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:07:58.723 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:07:58.724 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:07:58.724 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:07:58.724 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:07:58.724 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:07:58.724 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:07:59.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:07:59.227 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:07:59.228 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-26T00:07:59.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:07:59.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:07:59.228 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:07:59.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=523986010114 2024-03-26T00:07:59.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 523986010114 2024-03-26T00:07:59.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-523986010114' 2024-03-26T00:07:59.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:07:59.504 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:07:59.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=459561500681 2024-03-26T00:07:59.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 459561500681 2024-03-26T00:07:59.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-523986010114 1-459561500681' 2024-03-26T00:07:59.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:07:59.707 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:07:59.901 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318292 2024-03-26T00:07:59.901 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318292 2024-03-26T00:07:59.901 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-523986010114 1-459561500681 2-352187318292' 2024-03-26T00:07:59.901 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:07:59.902 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-523986010114 2024-03-26T00:07:59.902 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:07:59.902 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:07:59.903 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-523986010114 2024-03-26T00:07:59.903 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:07:59.905 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=523986010114 2024-03-26T00:07:59.905 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 523986010114' 2024-03-26T00:07:59.905 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 523986010114 2024-03-26T00:07:59.905 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:08:00.205 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 523986010114 2024-03-26T00:08:00.205 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:08:01.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:08:01.207 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:08:01.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 523986010114 -lt 523986010114 2024-03-26T00:08:01.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:08:01.509 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:08:01.509 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-459561500681 2024-03-26T00:08:01.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:08:01.511 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-459561500681 2024-03-26T00:08:01.512 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:08:01.514 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=459561500681 2024-03-26T00:08:01.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 459561500681' 2024-03-26T00:08:01.515 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 459561500681 2024-03-26T00:08:01.515 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:08:01.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 459561500681 -lt 459561500681 2024-03-26T00:08:01.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:08:01.830 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-352187318292 2024-03-26T00:08:01.830 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:08:01.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:08:01.832 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-352187318292 2024-03-26T00:08:01.832 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:08:01.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318292 2024-03-26T00:08:01.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 352187318292' 2024-03-26T00:08:01.834 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 352187318292 2024-03-26T00:08:01.834 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:08:02.126 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318292 -lt 352187318292 2024-03-26T00:08:02.127 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:08:02.127 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:08:02.127 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:08:02.491 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:08:02.491 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:08:02.491 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:08:02.492 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:08:02.492 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:08:02.492 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:08:02.492 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:08:02.492 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:08:02.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:08:02.804 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:08:02.804 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:08:02.804 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:08:03.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:08:03.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:08:03.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:08:03.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3670: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2024-03-26T00:08:03.167 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2024-03-26T00:08:03.168 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-26T00:08:03.168 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=EOBJ0 2024-03-26T00:08:03.168 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool EOBJ0 2024-03-26T00:08:03.168 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-26T00:08:03.471 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: local pg=3.0 2024-03-26T00:08:03.471 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3678: corrupt_scrub_erasure: pg_scrub 3.0 2024-03-26T00:08:03.471 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-03-26T00:08:03.472 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-03-26T00:08:03.472 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:08:03.472 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:08:03.473 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:08:03.473 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-26T00:08:03.837 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-03-26T00:05:20.416972+0000 2024-03-26T00:08:03.837 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-03-26T00:08:04.130 INFO:tasks.workunit.client.0.smithi129.stderr:instructing pg 3.0s0 on osd.1 to scrub 2024-03-26T00:08:04.148 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-03-26T00:05:20.416972+0000 2024-03-26T00:08:04.148 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-26T00:08:04.148 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-26T00:05:20.416972+0000 2024-03-26T00:08:04.149 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-26T00:08:04.149 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-26T00:08:04.149 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:08:04.149 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-26T00:08:04.149 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:08:04.149 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:08:04.149 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-26T00:08:04.149 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:08:04.440 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:05:20.416972+0000 '>' 2024-03-26T00:05:20.416972+0000 2024-03-26T00:08:04.440 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:08:05.441 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:08:05.441 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:08:05.442 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-26T00:08:05.442 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:08:05.442 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:08:05.442 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:08:05.443 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-26T00:08:05.762 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:05:20.416972+0000 '>' 2024-03-26T00:05:20.416972+0000 2024-03-26T00:08:05.762 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:08:06.763 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:08:06.764 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:08:06.765 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-26T00:08:06.765 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:08:06.765 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:08:06.766 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:08:06.766 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-26T00:08:07.126 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:05:20.416972+0000 '>' 2024-03-26T00:05:20.416972+0000 2024-03-26T00:08:07.126 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:08:08.128 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:08:08.128 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:08:08.129 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-26T00:08:08.130 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:08:08.130 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:08:08.130 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:08:08.131 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-26T00:08:08.439 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:08:04.810647+0000 '>' 2024-03-26T00:05:20.416972+0000 2024-03-26T00:08:08.439 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-26T00:08:08.439 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3680: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-03-26T00:08:08.511 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-03-26T00:08:08.538 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: test 1 = 1 2024-03-26T00:08:08.538 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-03-26T00:08:08.569 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: test 3.0 = 3.0 2024-03-26T00:08:08.569 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3686: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-03-26T00:08:08.598 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-03-26T00:08:08.625 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: epoch=122 2024-03-26T00:08:08.625 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq 'def walk(f): 2024-03-26T00:08:08.625 INFO:tasks.workunit.client.0.smithi129.stderr: . as $in 2024-03-26T00:08:08.625 INFO:tasks.workunit.client.0.smithi129.stderr: | if type == "object" then 2024-03-26T00:08:08.625 INFO:tasks.workunit.client.0.smithi129.stderr: reduce keys[] as $key 2024-03-26T00:08:08.625 INFO:tasks.workunit.client.0.smithi129.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-26T00:08:08.625 INFO:tasks.workunit.client.0.smithi129.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-26T00:08:08.626 INFO:tasks.workunit.client.0.smithi129.stderr: else f 2024-03-26T00:08:08.626 INFO:tasks.workunit.client.0.smithi129.stderr: end; 2024-03-26T00:08:08.626 INFO:tasks.workunit.client.0.smithi129.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-26T00:08:08.626 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-26T00:08:08.626 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-26T00:08:08.626 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-26T00:08:08.626 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-03-26T00:08:08.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq .inconsistents 2024-03-26T00:08:08.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-26T00:08:08.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq 'def walk(f): 2024-03-26T00:08:08.667 INFO:tasks.workunit.client.0.smithi129.stderr: . as $in 2024-03-26T00:08:08.667 INFO:tasks.workunit.client.0.smithi129.stderr: | if type == "object" then 2024-03-26T00:08:08.667 INFO:tasks.workunit.client.0.smithi129.stderr: reduce keys[] as $key 2024-03-26T00:08:08.667 INFO:tasks.workunit.client.0.smithi129.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-26T00:08:08.667 INFO:tasks.workunit.client.0.smithi129.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-26T00:08:08.667 INFO:tasks.workunit.client.0.smithi129.stderr: else f 2024-03-26T00:08:08.667 INFO:tasks.workunit.client.0.smithi129.stderr: end; 2024-03-26T00:08:08.667 INFO:tasks.workunit.client.0.smithi129.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-26T00:08:08.667 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-26T00:08:08.667 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-26T00:08:08.667 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-26T00:08:08.668 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-03-26T00:08:08.668 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq .inconsistents 2024-03-26T00:08:08.668 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-26T00:08:08.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4311: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-26T00:08:08.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-26T00:08:08.708 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4312: corrupt_scrub_erasure: test no = yes 2024-03-26T00:08:08.708 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4317: corrupt_scrub_erasure: test '' = yes 2024-03-26T00:08:08.708 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4322: corrupt_scrub_erasure: pg_deep_scrub 3.0 2024-03-26T00:08:08.708 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-03-26T00:08:08.709 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-26T00:08:08.709 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:08:08.709 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-26T00:08:08.710 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-26T00:08:08.710 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:08:09.035 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-03-26T00:05:20.416972+0000 2024-03-26T00:08:09.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-03-26T00:08:09.319 INFO:tasks.workunit.client.0.smithi129.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2024-03-26T00:08:09.335 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-03-26T00:05:20.416972+0000 last_deep_scrub_stamp 2024-03-26T00:08:09.336 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-26T00:08:09.336 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-26T00:05:20.416972+0000 2024-03-26T00:08:09.336 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-03-26T00:08:09.336 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-26T00:08:09.336 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:08:09.337 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-26T00:08:09.337 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:08:09.337 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-26T00:08:09.337 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:08:09.337 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-26T00:08:09.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:05:20.416972+0000 '>' 2024-03-26T00:05:20.416972+0000 2024-03-26T00:08:09.635 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:08:10.636 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:08:10.636 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:08:10.637 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-26T00:08:10.637 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:08:10.637 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-26T00:08:10.637 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:08:10.637 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-26T00:08:10.943 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:05:20.416972+0000 '>' 2024-03-26T00:05:20.416972+0000 2024-03-26T00:08:10.943 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:08:11.944 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:08:11.945 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:08:11.945 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-26T00:08:11.945 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:08:11.945 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-26T00:08:11.946 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:08:11.946 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-26T00:08:12.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:08:10.042091+0000 '>' 2024-03-26T00:05:20.416972+0000 2024-03-26T00:08:12.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-26T00:08:12.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4324: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-03-26T00:08:12.340 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-03-26T00:08:12.367 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: test 1 = 1 2024-03-26T00:08:12.367 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-03-26T00:08:12.398 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: test 3.0 = 3.0 2024-03-26T00:08:12.399 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4330: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-03-26T00:08:12.430 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-03-26T00:08:12.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: epoch=122 2024-03-26T00:08:12.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4334: corrupt_scrub_erasure: '[' false = true ']' 2024-03-26T00:08:12.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: jq .inconsistents 2024-03-26T00:08:12.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: jq 'def walk(f): 2024-03-26T00:08:12.460 INFO:tasks.workunit.client.0.smithi129.stderr: . as $in 2024-03-26T00:08:12.460 INFO:tasks.workunit.client.0.smithi129.stderr: | if type == "object" then 2024-03-26T00:08:12.460 INFO:tasks.workunit.client.0.smithi129.stderr: reduce keys[] as $key 2024-03-26T00:08:12.460 INFO:tasks.workunit.client.0.smithi129.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-26T00:08:12.460 INFO:tasks.workunit.client.0.smithi129.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-26T00:08:12.460 INFO:tasks.workunit.client.0.smithi129.stderr: else f 2024-03-26T00:08:12.460 INFO:tasks.workunit.client.0.smithi129.stderr: end; 2024-03-26T00:08:12.461 INFO:tasks.workunit.client.0.smithi129.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-26T00:08:12.461 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-26T00:08:12.461 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-26T00:08:12.461 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-26T00:08:12.461 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-03-26T00:08:12.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-26T00:08:12.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq 'def walk(f): 2024-03-26T00:08:12.509 INFO:tasks.workunit.client.0.smithi129.stderr: . as $in 2024-03-26T00:08:12.509 INFO:tasks.workunit.client.0.smithi129.stderr: | if type == "object" then 2024-03-26T00:08:12.509 INFO:tasks.workunit.client.0.smithi129.stderr: reduce keys[] as $key 2024-03-26T00:08:12.509 INFO:tasks.workunit.client.0.smithi129.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-26T00:08:12.509 INFO:tasks.workunit.client.0.smithi129.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-26T00:08:12.509 INFO:tasks.workunit.client.0.smithi129.stderr: else f 2024-03-26T00:08:12.509 INFO:tasks.workunit.client.0.smithi129.stderr: end; 2024-03-26T00:08:12.509 INFO:tasks.workunit.client.0.smithi129.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-26T00:08:12.509 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-26T00:08:12.509 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-26T00:08:12.509 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-26T00:08:12.509 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-03-26T00:08:12.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq .inconsistents 2024-03-26T00:08:12.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-26T00:08:12.552 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5725: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-26T00:08:12.552 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-26T00:08:12.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5726: corrupt_scrub_erasure: test no = yes 2024-03-26T00:08:12.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5737: corrupt_scrub_erasure: test '' = yes 2024-03-26T00:08:12.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5742: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2024-03-26T00:08:13.006 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'ecpool' removed 2024-03-26T00:08:13.030 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-26T00:08:13.030 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-26T00:08:13.030 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-26T00:08:13.030 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-26T00:08:13.031 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:08:13.031 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:08:13.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:08:13.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:08:13.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:08:13.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:08:13.346 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-26T00:08:13.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-26T00:08:13.348 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-26T00:08:13.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-26T00:08:13.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-26T00:08:13.350 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-26T00:08:13.351 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:08:13.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-26T00:08:13.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-26T00:08:13.353 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:08:13.354 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-26T00:08:13.355 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-26T00:08:13.355 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-26T00:08:13.393 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-26T00:08:13.393 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:08:13.393 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:08:13.393 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-26T00:08:13.396 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-26T00:08:13.396 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-26T00:08:13.396 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-26T00:08:13.396 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-26T00:08:13.396 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-26T00:08:13.396 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-26T00:08:13.396 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-26T00:08:13.396 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-26T00:08:13.396 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-26T00:08:13.397 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:08:13.397 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:08:13.397 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:08:13.397 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:08:13.397 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:08:13.400 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:08:13.400 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-26T00:08:13.402 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-26T00:08:13.402 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-26T00:08:13.403 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-26T00:08:13.404 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-26T00:08:13.404 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-26T00:08:13.404 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:08:13.405 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-26T00:08:13.405 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-26T00:08:13.405 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:08:13.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-26T00:08:13.408 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-26T00:08:13.408 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-26T00:08:13.409 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-26T00:08:13.410 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:08:13.410 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:08:13.410 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-26T00:08:13.411 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-26T00:08:13.411 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-26T00:08:13.411 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-26T00:08:13.412 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-26T00:08:13.412 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:08:13.413 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:08:13.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61636 2024-03-26T00:08:13.416 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-26T00:08:13.416 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-26T00:08:13.416 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-26T00:08:13.416 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-26T00:08:13.416 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_erasure_overwrites td/osd-scrub-repair 2024-03-26T00:08:13.416 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5750: TEST_corrupt_scrub_erasure_overwrites: '[' true = true ']' 2024-03-26T00:08:13.416 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5751: TEST_corrupt_scrub_erasure_overwrites: corrupt_scrub_erasure td/osd-scrub-repair true 2024-03-26T00:08:13.416 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3596: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2024-03-26T00:08:13.416 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3597: corrupt_scrub_erasure: local allow_overwrites=true 2024-03-26T00:08:13.416 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3598: corrupt_scrub_erasure: local poolname=ecpool 2024-03-26T00:08:13.416 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3599: corrupt_scrub_erasure: local total_objs=7 2024-03-26T00:08:13.416 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3601: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2024-03-26T00:08:13.417 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-26T00:08:13.417 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-26T00:08:13.417 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-26T00:08:13.417 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-26T00:08:13.417 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-26T00:08:13.417 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-26T00:08:13.883 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-26T00:08:13.883 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:08:13.883 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:08:13.884 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:08:13.884 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:08:13.884 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:08:13.884 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:08:13.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-26T00:08:14.103 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-26T00:08:14.104 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-26T00:08:14.104 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-26T00:08:14.104 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-26T00:08:14.104 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-26T00:08:14.105 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-26T00:08:14.105 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-26T00:08:14.105 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-26T00:08:14.105 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-26T00:08:14.106 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-26T00:08:14.106 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:08:14.106 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:08:14.106 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-26T00:08:14.106 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-26T00:08:14.106 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get fsid 2024-03-26T00:08:14.217 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-26T00:08:14.218 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-26T00:08:14.218 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-26T00:08:14.218 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-26T00:08:14.219 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-26T00:08:14.219 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-26T00:08:14.219 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-26T00:08:14.219 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-26T00:08:14.219 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-26T00:08:14.219 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:08:14.219 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:08:14.220 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-26T00:08:14.220 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-26T00:08:14.220 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get mon_host 2024-03-26T00:08:14.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3602: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2024-03-26T00:08:14.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-26T00:08:14.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-26T00:08:14.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-26T00:08:14.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-26T00:08:14.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-26T00:08:14.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-26T00:08:15.200 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-26T00:08:15.201 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:08:15.201 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:08:15.201 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:08:15.201 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:08:15.201 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:08:15.202 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:08:15.203 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-26T00:08:15.204 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-26T00:08:15.240 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: seq 0 2 2024-03-26T00:08:15.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-03-26T00:08:15.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 0 2024-03-26T00:08:15.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-26T00:08:15.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-26T00:08:15.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-26T00:08:15.244 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-26T00:08:15.244 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:08:15.244 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:08:15.244 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:08:15.244 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:08:15.244 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:08:15.244 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-26T00:08:15.245 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-26T00:08:15.245 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-26T00:08:15.245 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-26T00:08:15.245 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:08:15.245 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-26T00:08:15.245 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:08:15.245 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:08:15.245 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:08:15.245 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:08:15.245 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:08:15.245 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:08:15.245 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:08:15.245 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:08:15.245 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-26T00:08:15.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-26T00:08:15.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:08:15.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:08:15.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:08:15.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:08:15.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:08:15.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:08:15.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-26T00:08:15.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-26T00:08:15.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-26T00:08:15.247 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-26T00:08:15.247 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=27293645-99f8-4138-92f4-30c701220322 2024-03-26T00:08:15.247 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 27293645-99f8-4138-92f4-30c701220322' 2024-03-26T00:08:15.247 INFO:tasks.workunit.client.0.smithi129.stdout:add osd0 27293645-99f8-4138-92f4-30c701220322 2024-03-26T00:08:15.248 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-26T00:08:15.264 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDvEQJmCfzaDxAARuxrziiyMKBXIbHuQ8Ex6w== 2024-03-26T00:08:15.264 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDvEQJmCfzaDxAARuxrziiyMKBXIbHuQ8Ex6w=="}' 2024-03-26T00:08:15.264 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 27293645-99f8-4138-92f4-30c701220322 -i td/osd-scrub-repair/0/new.json 2024-03-26T00:08:15.498 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-26T00:08:15.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-26T00:08:15.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDvEQJmCfzaDxAARuxrziiyMKBXIbHuQ8Ex6w== --osd-uuid 27293645-99f8-4138-92f4-30c701220322 2024-03-26T00:08:15.549 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:08:15.548+0000 7f09142547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:08:15.550 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:08:15.552+0000 7f09142547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:08:15.552 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:08:15.552+0000 7f09142547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:08:15.552 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:08:15.552+0000 7f09142547c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-26T00:08:15.552 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:08:15.552+0000 7f09142547c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-26T00:08:15.552 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:08:15.552+0000 7f09142547c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-26T00:08:15.640 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:08:15.640+0000 7f09142547c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-26T00:08:21.823 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-26T00:08:21.824 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-26T00:08:21.825 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd0 key to auth repository 2024-03-26T00:08:21.826 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-26T00:08:21.826 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-26T00:08:22.223 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.0 2024-03-26T00:08:22.224 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-26T00:08:22.224 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:08:22.224 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-26T00:08:22.224 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-26T00:08:22.224 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-26T00:08:22.250 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:08:22.247+0000 7f20680367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:08:22.251 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:08:22.251+0000 7f20680367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:08:22.252 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:08:22.251+0000 7f20680367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:08:22.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-26T00:08:22.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:08:22.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-26T00:08:22.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:08:22.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:08:22.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:08:22.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:08:22.572 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-26T00:08:22.573 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:08:22.573 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:08:22.796 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:08:22.795+0000 7f20680367c0 -1 Falling back to public interface 2024-03-26T00:08:22.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:08:23.913 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:08:23.913 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:08:23.913 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-26T00:08:23.913 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:08:23.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:08:23.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:08:24.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:08:25.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:08:25.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:08:25.229 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-26T00:08:25.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:08:25.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:08:25.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:08:25.416 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:08:25.415+0000 7f20680367c0 -1 osd.0 0 log_to_monitors true 2024-03-26T00:08:25.549 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:08:26.551 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:08:26.551 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:08:26.551 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-26T00:08:26.552 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:08:26.552 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:08:26.552 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:08:26.878 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:08:27.879 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:08:27.880 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:08:27.880 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-26T00:08:27.880 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-26T00:08:27.881 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:08:27.881 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:08:28.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:08:29.247 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:08:29.248 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:08:29.248 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-26T00:08:29.248 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-26T00:08:29.248 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:08:29.249 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:08:29.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:08:30.563 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:08:30.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:08:30.564 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-26T00:08:30.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-26T00:08:30.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:08:30.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:08:30.875 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:08:31.876 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:08:31.877 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:08:31.877 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-26T00:08:31.877 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-26T00:08:31.878 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:08:31.878 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:08:32.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:08:33.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:08:33.266 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:08:33.266 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-26T00:08:33.266 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-26T00:08:33.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:08:33.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:08:33.591 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:08:34.592 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:08:34.592 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:08:34.593 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-26T00:08:34.593 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-26T00:08:34.593 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:08:34.593 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:08:34.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:08:35.916 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:08:35.916 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:08:35.916 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-26T00:08:35.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-26T00:08:35.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:08:35.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:08:36.235 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:08:37.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:08:37.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:08:37.237 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-26T00:08:37.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-26T00:08:37.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:08:37.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:08:37.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:08:38.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:08:38.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:08:38.562 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-26T00:08:38.563 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-26T00:08:38.563 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:08:38.563 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:08:38.876 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:08:39.878 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:08:39.878 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:08:39.879 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-26T00:08:39.879 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-26T00:08:39.879 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:08:39.879 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:08:40.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:08:41.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:08:41.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:08:41.283 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-26T00:08:41.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-26T00:08:41.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:08:41.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:08:41.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:08:42.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:08:42.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:08:42.783 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-26T00:08:42.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-26T00:08:42.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:08:42.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:08:43.339 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:08:44.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:08:44.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:08:44.341 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-26T00:08:44.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-26T00:08:44.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:08:44.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:08:44.719 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:08:45.721 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:08:45.721 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:08:45.722 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-26T00:08:45.722 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-26T00:08:45.723 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:08:45.723 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:08:46.270 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:08:47.272 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:08:47.272 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:08:47.273 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-26T00:08:47.273 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-26T00:08:47.273 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:08:47.274 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:08:48.091 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:08:49.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:08:49.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:08:49.094 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-26T00:08:49.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-26T00:08:49.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:08:49.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:08:49.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:08:50.422 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:08:50.423 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:08:50.423 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-26T00:08:50.423 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-26T00:08:50.423 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:08:50.424 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:08:50.750 INFO:tasks.workunit.client.0.smithi129.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/755768703,v1:127.0.0.1:6803/755768703] [v2:127.0.0.1:6804/755768703,v1:127.0.0.1:6805/755768703] exists,up 27293645-99f8-4138-92f4-30c701220322 2024-03-26T00:08:50.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:08:50.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:08:50.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:08:50.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-03-26T00:08:50.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 1 2024-03-26T00:08:50.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-26T00:08:50.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-26T00:08:50.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-26T00:08:50.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-26T00:08:50.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:08:50.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:08:50.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:08:50.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:08:50.754 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:08:50.754 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:08:50.754 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:08:50.754 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-26T00:08:50.754 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-26T00:08:50.754 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:08:50.754 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-26T00:08:50.754 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:08:50.754 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:08:50.756 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:08:50.756 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:08:50.756 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:08:50.756 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:08:50.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:08:50.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:08:50.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-26T00:08:50.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-26T00:08:50.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:08:50.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:08:50.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:08:50.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:08:50.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:08:50.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:08:50.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-26T00:08:50.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-26T00:08:50.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:08:50.757 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-26T00:08:50.757 INFO:tasks.workunit.client.0.smithi129.stdout:add osd1 4d7ea6fb-1a4b-4612-ace7-7c4560a2fc00 2024-03-26T00:08:50.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4d7ea6fb-1a4b-4612-ace7-7c4560a2fc00 2024-03-26T00:08:50.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 4d7ea6fb-1a4b-4612-ace7-7c4560a2fc00' 2024-03-26T00:08:50.758 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-26T00:08:50.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQASEgJmI7OeLhAAlrrjZDkEG5Tkzj0thdF9JQ== 2024-03-26T00:08:50.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQASEgJmI7OeLhAAlrrjZDkEG5Tkzj0thdF9JQ=="}' 2024-03-26T00:08:50.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4d7ea6fb-1a4b-4612-ace7-7c4560a2fc00 -i td/osd-scrub-repair/1/new.json 2024-03-26T00:08:51.156 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-26T00:08:51.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-26T00:08:51.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQASEgJmI7OeLhAAlrrjZDkEG5Tkzj0thdF9JQ== --osd-uuid 4d7ea6fb-1a4b-4612-ace7-7c4560a2fc00 2024-03-26T00:08:51.209 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:08:51.207+0000 7f4476e827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:08:51.267 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:08:51.267+0000 7f4476e827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:08:51.268 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:08:51.267+0000 7f4476e827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:08:51.269 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:08:51.267+0000 7f4476e827c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-26T00:08:51.269 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:08:51.267+0000 7f4476e827c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-26T00:08:51.269 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:08:51.267+0000 7f4476e827c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-26T00:08:51.342 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:08:51.343+0000 7f4476e827c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-26T00:08:57.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-26T00:08:57.909 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-26T00:08:57.910 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd1 key to auth repository 2024-03-26T00:08:57.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-26T00:08:57.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-26T00:08:58.298 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.1 2024-03-26T00:08:58.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-26T00:08:58.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:08:58.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-26T00:08:58.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-26T00:08:58.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-26T00:08:58.324 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:08:58.323+0000 7f85b7afb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:08:58.329 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:08:58.327+0000 7f85b7afb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:08:58.329 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:08:58.327+0000 7f85b7afb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:08:58.629 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-26T00:08:58.629 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:08:58.629 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:08:58.629 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:08:58.629 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:08:58.629 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:08:58.629 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-26T00:08:58.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:08:58.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:08:58.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:08:58.892 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:08:58.891+0000 7f85b7afb7c0 -1 Falling back to public interface 2024-03-26T00:08:58.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:08:59.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:08:59.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:08:59.965 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-26T00:08:59.965 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:08:59.965 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:08:59.965 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:09:00.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:09:01.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:09:01.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:09:01.279 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-26T00:09:01.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:09:01.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:09:01.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:09:01.537 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:09:01.535+0000 7f85b7afb7c0 -1 osd.1 0 log_to_monitors true 2024-03-26T00:09:01.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:09:02.601 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:09:02.602 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:09:02.602 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-26T00:09:02.602 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:09:02.603 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:09:02.603 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:09:02.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:09:03.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:09:03.924 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:09:03.924 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-26T00:09:03.924 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-26T00:09:03.925 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:09:03.925 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:09:04.248 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:09:05.249 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:09:05.250 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:09:05.250 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-26T00:09:05.250 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-26T00:09:05.251 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:09:05.251 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:09:05.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:09:06.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:09:06.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:09:06.650 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-26T00:09:06.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-26T00:09:06.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:09:06.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:09:07.115 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:09:08.116 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:09:08.117 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:09:08.117 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-26T00:09:08.117 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-26T00:09:08.117 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:09:08.118 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:09:08.429 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:09:09.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:09:09.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:09:09.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-26T00:09:09.432 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-26T00:09:09.432 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:09:09.432 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:09:09.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:09:10.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:09:10.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:09:10.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-26T00:09:10.742 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-26T00:09:10.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:09:10.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:09:11.051 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:09:12.053 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:09:12.053 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:09:12.053 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-26T00:09:12.054 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-26T00:09:12.054 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:09:12.054 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:09:12.440 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:09:13.442 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:09:13.442 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:09:13.443 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-26T00:09:13.443 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-26T00:09:13.443 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:09:13.443 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:09:13.754 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:09:14.754 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:09:14.754 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:09:14.754 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-26T00:09:14.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-26T00:09:14.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:09:14.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:09:15.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:09:16.068 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:09:16.068 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:09:16.068 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-26T00:09:16.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-26T00:09:16.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:09:16.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:09:16.566 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:09:17.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:09:17.569 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:09:17.569 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-26T00:09:17.569 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-26T00:09:17.569 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:09:17.570 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:09:18.063 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:09:19.065 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:09:19.065 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:09:19.066 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-26T00:09:19.066 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-26T00:09:19.066 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:09:19.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:09:19.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:09:20.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:09:20.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:09:20.897 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-26T00:09:20.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-26T00:09:20.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:09:20.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:09:21.451 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:09:22.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:09:22.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:09:22.453 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-26T00:09:22.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-26T00:09:22.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:09:22.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:09:23.244 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:09:24.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:09:24.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:09:24.246 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-26T00:09:24.247 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-26T00:09:24.247 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:09:24.247 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:09:24.970 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:09:25.971 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:09:25.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:09:25.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-26T00:09:25.972 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-26T00:09:25.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:09:25.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:09:26.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:09:27.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:09:27.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:09:27.434 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-26T00:09:27.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-26T00:09:27.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:09:27.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:09:27.750 INFO:tasks.workunit.client.0.smithi129.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3965569623,v1:127.0.0.1:6811/3965569623] [v2:127.0.0.1:6812/3965569623,v1:127.0.0.1:6813/3965569623] exists,up 4d7ea6fb-1a4b-4612-ace7-7c4560a2fc00 2024-03-26T00:09:27.754 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:09:27.754 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:09:27.754 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:09:27.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-03-26T00:09:27.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 2 2024-03-26T00:09:27.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-26T00:09:27.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-26T00:09:27.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-26T00:09:27.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-26T00:09:27.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-26T00:09:27.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:09:27.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:09:27.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:09:27.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:09:27.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-26T00:09:27.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-26T00:09:27.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-26T00:09:27.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-26T00:09:27.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:09:27.756 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-26T00:09:27.756 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:09:27.756 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:09:27.757 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:09:27.757 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:09:27.757 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:09:27.757 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:09:27.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:09:27.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:09:27.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-26T00:09:27.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-26T00:09:27.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:09:27.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:09:27.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:09:27.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:09:27.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:09:27.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:09:27.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-26T00:09:27.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-26T00:09:27.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-26T00:09:27.758 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-26T00:09:27.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8d661fa8-dc51-4e46-8532-fc7445949179 2024-03-26T00:09:27.759 INFO:tasks.workunit.client.0.smithi129.stdout:add osd2 8d661fa8-dc51-4e46-8532-fc7445949179 2024-03-26T00:09:27.759 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 8d661fa8-dc51-4e46-8532-fc7445949179' 2024-03-26T00:09:27.759 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-26T00:09:27.773 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA3EgJmxrQ9LhAArj5Ju0ZXisyU92DOmZOXYQ== 2024-03-26T00:09:27.773 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA3EgJmxrQ9LhAArj5Ju0ZXisyU92DOmZOXYQ=="}' 2024-03-26T00:09:27.773 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8d661fa8-dc51-4e46-8532-fc7445949179 -i td/osd-scrub-repair/2/new.json 2024-03-26T00:09:28.178 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-26T00:09:28.195 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-26T00:09:28.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA3EgJmxrQ9LhAArj5Ju0ZXisyU92DOmZOXYQ== --osd-uuid 8d661fa8-dc51-4e46-8532-fc7445949179 2024-03-26T00:09:28.232 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:09:28.230+0000 7efd675677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:09:28.280 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:09:28.278+0000 7efd675677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:09:28.281 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:09:28.278+0000 7efd675677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:09:28.282 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:09:28.282+0000 7efd675677c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-26T00:09:28.282 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:09:28.282+0000 7efd675677c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-26T00:09:28.282 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:09:28.282+0000 7efd675677c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-26T00:09:28.351 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:09:28.350+0000 7efd675677c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-26T00:09:35.131 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-26T00:09:35.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-26T00:09:35.133 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd2 key to auth repository 2024-03-26T00:09:35.133 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-26T00:09:35.134 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-26T00:09:35.521 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.2 2024-03-26T00:09:35.521 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-26T00:09:35.522 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:09:35.522 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-26T00:09:35.522 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-26T00:09:35.522 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-26T00:09:35.552 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:09:35.550+0000 7fae2f0d27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:09:35.554 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:09:35.554+0000 7fae2f0d27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:09:35.555 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:09:35.554+0000 7fae2f0d27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:09:35.865 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-26T00:09:35.865 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:09:35.865 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-26T00:09:35.865 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:09:35.865 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:09:35.866 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:09:35.866 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:09:35.866 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-26T00:09:35.866 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:09:35.866 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:09:36.104 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:09:36.102+0000 7fae2f0d27c0 -1 Falling back to public interface 2024-03-26T00:09:36.205 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:09:37.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:09:37.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:09:37.208 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-26T00:09:37.208 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:09:37.208 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:09:37.208 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:09:37.524 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:09:38.526 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:09:38.526 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:09:38.526 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-26T00:09:38.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:09:38.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:09:38.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:09:38.751 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:09:38.750+0000 7fae2f0d27c0 -1 osd.2 0 log_to_monitors true 2024-03-26T00:09:38.867 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:09:39.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:09:39.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:09:39.869 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-26T00:09:39.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:09:39.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:09:39.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:09:40.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:09:41.196 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:09:41.196 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:09:41.197 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-26T00:09:41.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-26T00:09:41.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:09:41.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:09:41.519 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:09:42.522 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:09:42.522 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:09:42.522 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-26T00:09:42.522 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-26T00:09:42.523 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:09:42.523 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:09:42.906 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:09:43.909 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:09:43.909 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:09:43.909 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-26T00:09:43.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-26T00:09:43.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:09:43.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:09:44.226 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:09:45.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:09:45.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:09:45.228 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-26T00:09:45.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-26T00:09:45.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:09:45.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:09:45.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:09:46.547 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:09:46.548 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:09:46.548 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-26T00:09:46.548 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-26T00:09:46.548 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:09:46.548 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:09:46.855 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:09:47.857 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:09:47.857 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:09:47.857 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-26T00:09:47.858 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-26T00:09:47.858 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:09:47.858 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:09:48.158 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:09:49.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:09:49.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:09:49.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-26T00:09:49.161 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-26T00:09:49.161 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:09:49.161 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:09:49.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:09:50.469 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:09:50.469 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:09:50.470 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-26T00:09:50.470 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-26T00:09:50.471 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:09:50.471 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:09:50.825 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:09:51.827 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:09:51.827 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:09:51.827 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-26T00:09:51.828 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-26T00:09:51.828 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:09:51.828 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:09:52.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:09:53.144 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:09:53.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:09:53.145 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-26T00:09:53.146 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-26T00:09:53.146 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:09:53.146 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:09:53.479 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:09:54.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:09:54.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:09:54.481 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-26T00:09:54.482 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-26T00:09:54.482 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:09:54.482 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:09:55.288 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:09:56.289 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:09:56.290 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:09:56.290 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-26T00:09:56.290 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-26T00:09:56.291 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:09:56.291 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:09:57.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:09:58.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:09:58.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:09:58.162 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-26T00:09:58.163 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-26T00:09:58.163 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:09:58.163 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:09:59.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:10:00.025 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:10:00.026 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:10:00.026 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-26T00:10:00.026 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-26T00:10:00.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:10:00.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:10:00.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:10:01.786 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:10:01.786 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:10:01.786 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-26T00:10:01.787 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-26T00:10:01.787 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:10:01.787 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:10:02.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:10:03.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:10:03.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:10:03.177 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-26T00:10:03.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-26T00:10:03.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:10:03.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:10:03.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:10:04.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:10:04.656 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:10:04.657 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-26T00:10:04.657 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-26T00:10:04.658 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:10:04.658 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:10:04.971 INFO:tasks.workunit.client.0.smithi129.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3844040758,v1:127.0.0.1:6819/3844040758] [v2:127.0.0.1:6820/3844040758,v1:127.0.0.1:6821/3844040758] exists,up 8d661fa8-dc51-4e46-8532-fc7445949179 2024-03-26T00:10:04.971 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:10:04.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:10:04.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:10:04.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3606: corrupt_scrub_erasure: create_rbd_pool 2024-03-26T00:10:04.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-26T00:10:05.267 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'rbd' does not exist 2024-03-26T00:10:05.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-26T00:10:05.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-26T00:10:05.700 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'rbd' created 2024-03-26T00:10:05.717 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-26T00:10:06.719 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-26T00:10:08.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3607: corrupt_scrub_erasure: create_pool foo 1 2024-03-26T00:10:08.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-03-26T00:10:09.368 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'foo' created 2024-03-26T00:10:09.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-26T00:10:10.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3609: corrupt_scrub_erasure: create_ec_pool ecpool true k=2 m=1 stripe_unit=2K --force 2024-03-26T00:10:10.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-26T00:10:10.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-26T00:10:10.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-03-26T00:10:10.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-26T00:10:10.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2024-03-26T00:10:10.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-26T00:10:10.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-26T00:10:11.578 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'ecpool' created 2024-03-26T00:10:11.600 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-26T00:10:12.602 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-03-26T00:10:12.602 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-03-26T00:10:13.192 INFO:tasks.workunit.client.0.smithi129.stderr:set pool 3 allow_ec_overwrites to true 2024-03-26T00:10:13.213 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-26T00:10:13.213 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:10:13.213 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:10:13.213 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:10:13.214 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:10:13.215 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:10:13.215 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:10:13.215 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:10:13.215 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:10:13.215 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:10:13.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:10:13.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:10:13.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:10:13.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:10:13.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:10:13.331 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:10:13.632 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:10:13.633 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:10:13.633 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-26T00:10:13.633 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:10:13.633 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:10:13.633 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:10:13.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836501 2024-03-26T00:10:13.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836501 2024-03-26T00:10:13.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501' 2024-03-26T00:10:13.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:10:13.777 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:10:13.975 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672973 2024-03-26T00:10:13.975 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672973 2024-03-26T00:10:13.975 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-42949672973' 2024-03-26T00:10:13.976 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:10:13.976 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:10:14.133 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509451 2024-03-26T00:10:14.133 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509451 2024-03-26T00:10:14.133 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-42949672973 2-64424509451' 2024-03-26T00:10:14.133 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:10:14.133 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836501 2024-03-26T00:10:14.134 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:10:14.134 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:10:14.134 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836501 2024-03-26T00:10:14.134 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:10:14.134 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836501 2024-03-26T00:10:14.134 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-03-26T00:10:14.135 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 21474836501 2024-03-26T00:10:14.136 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:10:14.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836501 2024-03-26T00:10:14.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:10:15.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:10:15.462 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:10:15.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836501 2024-03-26T00:10:15.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:10:16.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-26T00:10:16.780 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:10:17.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836501 2024-03-26T00:10:17.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:10:17.096 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672973 2024-03-26T00:10:17.096 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:10:17.098 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:10:17.099 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672973 2024-03-26T00:10:17.099 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:10:17.101 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672973 2024-03-26T00:10:17.101 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 42949672973 2024-03-26T00:10:17.101 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-03-26T00:10:17.101 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:10:17.432 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672973 -lt 42949672973 2024-03-26T00:10:17.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:10:17.433 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:10:17.434 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509451 2024-03-26T00:10:17.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:10:17.436 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509451 2024-03-26T00:10:17.436 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:10:17.438 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509451 2024-03-26T00:10:17.438 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 64424509451 2024-03-26T00:10:17.438 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-03-26T00:10:17.439 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:10:17.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509452 -lt 64424509451 2024-03-26T00:10:17.762 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:10:17.763 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:10:17.763 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:10:18.105 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:10:18.105 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:10:18.106 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:10:18.106 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:10:18.106 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:10:18.106 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:10:18.106 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:10:18.106 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:10:18.438 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:10:18.439 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:10:18.439 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:10:18.439 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:10:18.801 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:10:18.801 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:10:18.801 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:10:18.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-26T00:10:18.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3610: corrupt_scrub_erasure: wait_for_clean 2024-03-26T00:10:18.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:10:18.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:10:18.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:10:18.803 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:10:18.803 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:10:18.803 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:10:18.803 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:10:18.803 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:10:18.803 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:10:18.909 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:10:18.909 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:10:18.909 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:10:18.909 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:10:18.909 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:10:18.909 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:10:19.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:10:19.225 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:10:19.225 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-26T00:10:19.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:10:19.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:10:19.226 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:10:19.410 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836503 2024-03-26T00:10:19.410 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836503 2024-03-26T00:10:19.410 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503' 2024-03-26T00:10:19.410 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:10:19.410 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:10:19.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672975 2024-03-26T00:10:19.573 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672975 2024-03-26T00:10:19.573 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672975' 2024-03-26T00:10:19.573 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:10:19.573 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:10:19.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-03-26T00:10:19.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-03-26T00:10:19.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672975 2-64424509453' 2024-03-26T00:10:19.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:10:19.713 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836503 2024-03-26T00:10:19.713 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:10:19.715 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:10:19.715 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836503 2024-03-26T00:10:19.715 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:10:19.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836503 2024-03-26T00:10:19.716 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 21474836503 2024-03-26T00:10:19.718 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-03-26T00:10:19.718 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:10:20.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836503 2024-03-26T00:10:20.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:10:20.011 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:10:20.011 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672975 2024-03-26T00:10:20.012 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:10:20.014 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672975 2024-03-26T00:10:20.014 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:10:20.014 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672975 2024-03-26T00:10:20.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-03-26T00:10:20.015 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 42949672975 2024-03-26T00:10:20.015 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:10:20.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672975 -lt 42949672975 2024-03-26T00:10:20.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:10:20.330 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:10:20.330 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-03-26T00:10:20.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:10:20.332 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-03-26T00:10:20.332 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:10:20.333 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-03-26T00:10:20.334 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 64424509453 2024-03-26T00:10:20.334 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-03-26T00:10:20.334 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:10:20.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509453 -lt 64424509453 2024-03-26T00:10:20.634 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:10:20.635 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:10:20.635 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:10:21.018 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:10:21.018 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:10:21.019 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:10:21.019 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:10:21.019 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:10:21.019 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:10:21.019 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:10:21.019 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:10:21.361 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:10:21.362 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:10:21.362 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:10:21.363 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:10:21.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:10:21.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:10:21.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:10:21.714 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: seq 1 7 2024-03-26T00:10:21.715 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-26T00:10:21.715 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ1 2024-03-26T00:10:21.715 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2024-03-26T00:10:21.715 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-26T00:10:21.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-26T00:10:21.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ1 2024-03-26T00:10:21.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-26T00:10:21.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-26T00:10:21.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-26T00:10:22.188 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-26T00:10:22.209 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-26T00:10:22.773 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-26T00:10:22.792 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-26T00:10:22.793 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-26T00:10:22.793 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:10:23.114 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 1 % 2 2024-03-26T00:10:23.116 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-03-26T00:10:23.116 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-26T00:10:23.116 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3621: corrupt_scrub_erasure: local payload=UVWXYZZZ 2024-03-26T00:10:23.116 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3622: corrupt_scrub_erasure: echo UVWXYZZZ 2024-03-26T00:10:23.116 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3623: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-26T00:10:23.116 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:10:23.116 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:10:23.116 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-26T00:10:23.117 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:10:23.117 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-26T00:10:23.117 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:10:23.117 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:10:23.117 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-26T00:10:23.117 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:10:23.117 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-26T00:10:23.118 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:10:23.118 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:10:23.118 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:10:23.118 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:10:23.118 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:10:24.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:10:24.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-26T00:10:24.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:10:24.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:10:24.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:10:24.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:10:24.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:10:24.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-26T00:10:26.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-26T00:10:26.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:10:26.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:10:26.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-26T00:10:26.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:10:26.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:10:26.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:10:26.407 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:10:26.407 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:10:26.407 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:10:26.407 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:10:26.407 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:10:26.407 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:10:26.407 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:10:26.407 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:10:26.407 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:10:26.407 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:10:26.407 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:10:26.409 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:10:26.409 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:10:26.409 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:10:26.409 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:10:26.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:10:26.410 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:10:26.410 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:10:26.410 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:10:26.410 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:10:26.410 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:10:26.410 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:10:26.410 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:10:26.411 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:10:26.411 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:10:26.412 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:10:26.412 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-26T00:10:26.412 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.1 2024-03-26T00:10:26.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:10:26.413 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-26T00:10:26.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-26T00:10:26.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:10:26.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:10:26.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:10:26.444 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:10:26.445+0000 7f0da8d517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:10:26.446 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:10:26.449+0000 7f0da8d517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:10:26.448 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:10:26.449+0000 7f0da8d517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:10:26.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-26T00:10:26.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:10:26.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:10:26.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:10:26.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:10:26.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:10:26.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:10:26.732 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:10:26.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:10:26.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:10:26.981 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:10:26.981+0000 7f0da8d517c0 -1 Falling back to public interface 2024-03-26T00:10:27.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:10:28.082 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:10:28.082 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:10:28.082 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:10:28.083 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:10:28.083 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:10:28.083 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:10:28.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:10:29.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:10:29.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:10:29.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:10:29.409 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:10:29.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:10:29.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:10:29.718 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:10:29.717+0000 7f0da8d517c0 -1 osd.1 35 log_to_monitors true 2024-03-26T00:10:29.733 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:10:30.736 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:10:30.736 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:10:30.736 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:10:30.736 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:10:30.736 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:10:30.737 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:10:31.249 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1 up in weight 1 up_from 40 up_thru 40 down_at 36 last_clean_interval [10,35) [v2:127.0.0.1:6810/1279603354,v1:127.0.0.1:6811/1279603354] [v2:127.0.0.1:6812/1279603354,v1:127.0.0.1:6813/1279603354] exists,up 4d7ea6fb-1a4b-4612-ace7-7c4560a2fc00 2024-03-26T00:10:31.249 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:10:31.249 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:10:31.249 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:10:31.249 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:10:31.249 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:10:31.249 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:10:31.249 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:10:31.250 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:10:31.251 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:10:31.251 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:10:31.251 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:10:31.251 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:10:31.251 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:10:31.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:10:31.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:10:31.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:10:31.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:10:31.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:10:31.370 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:10:31.691 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:10:31.691 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:10:31.691 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-26T00:10:31.691 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:10:31.691 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:10:31.692 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:10:31.841 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836507 2024-03-26T00:10:31.841 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836507 2024-03-26T00:10:31.842 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507' 2024-03-26T00:10:31.842 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:10:31.842 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:10:31.985 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-03-26T00:10:31.986 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-03-26T00:10:31.986 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-171798691842' 2024-03-26T00:10:31.986 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:10:31.986 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:10:32.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509457 2024-03-26T00:10:32.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509457 2024-03-26T00:10:32.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-171798691842 2-64424509457' 2024-03-26T00:10:32.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:10:32.137 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836507 2024-03-26T00:10:32.137 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:10:32.137 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:10:32.138 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836507 2024-03-26T00:10:32.138 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:10:32.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836507 2024-03-26T00:10:32.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-03-26T00:10:32.140 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 21474836507 2024-03-26T00:10:32.140 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:10:32.442 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836507 2024-03-26T00:10:32.442 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:10:33.443 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:10:33.445 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:10:33.766 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836507 2024-03-26T00:10:33.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:10:34.768 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-26T00:10:34.769 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:10:35.092 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836507 -lt 21474836507 2024-03-26T00:10:35.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:10:35.093 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691842 2024-03-26T00:10:35.094 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:10:35.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:10:35.096 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691842 2024-03-26T00:10:35.096 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:10:35.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-03-26T00:10:35.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691842' 2024-03-26T00:10:35.098 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 171798691842 2024-03-26T00:10:35.098 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:10:35.436 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691842 -lt 171798691842 2024-03-26T00:10:35.437 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:10:35.437 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509457 2024-03-26T00:10:35.437 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:10:35.437 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:10:35.438 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509457 2024-03-26T00:10:35.438 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:10:35.439 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509457 2024-03-26T00:10:35.439 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-03-26T00:10:35.439 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 64424509457 2024-03-26T00:10:35.439 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:10:35.775 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509457 -lt 64424509457 2024-03-26T00:10:35.776 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:10:35.776 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:10:35.776 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:10:36.127 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:10:36.127 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:10:36.128 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:10:36.128 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:10:36.129 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:10:36.129 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:10:36.129 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:10:36.129 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:10:36.462 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:10:36.463 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:10:36.463 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:10:36.463 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:10:36.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:10:36.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:10:36.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:10:36.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-26T00:10:36.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ2 2024-03-26T00:10:36.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2024-03-26T00:10:36.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-26T00:10:36.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-26T00:10:36.821 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ2 2024-03-26T00:10:36.821 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-26T00:10:36.821 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-26T00:10:36.821 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-26T00:10:37.257 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-26T00:10:37.274 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-26T00:10:37.809 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-26T00:10:37.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-26T00:10:37.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-26T00:10:37.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:10:38.040 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 2 % 2 2024-03-26T00:10:38.041 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-03-26T00:10:38.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-26T00:10:38.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3628: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2024-03-26T00:10:38.045 INFO:tasks.workunit.client.0.smithi129.stderr:1+0 records in 2024-03-26T00:10:38.045 INFO:tasks.workunit.client.0.smithi129.stderr:1+0 records out 2024-03-26T00:10:38.045 INFO:tasks.workunit.client.0.smithi129.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 0.000160295 s, 12.8 MB/s 2024-03-26T00:10:38.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-26T00:10:38.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:10:38.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:10:38.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-26T00:10:38.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:10:38.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-26T00:10:38.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:10:38.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:10:38.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-26T00:10:38.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:10:38.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-26T00:10:38.046 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:10:38.046 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:10:38.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:10:38.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:10:38.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:10:39.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:10:39.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-26T00:10:39.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:10:39.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:10:39.360 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:10:39.360 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:10:39.360 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:10:39.360 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-26T00:10:41.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-26T00:10:41.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:10:41.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:10:41.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-26T00:10:41.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:10:41.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:10:41.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:10:41.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:10:41.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:10:41.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:10:41.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-26T00:10:41.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-26T00:10:41.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:10:41.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:10:41.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:10:41.435 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:10:41.435 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:10:41.435 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:10:41.436 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:10:41.436 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:10:41.436 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:10:41.437 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:10:41.437 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:10:41.437 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:10:41.437 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:10:41.437 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:10:41.437 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:10:41.437 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:10:41.437 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:10:41.437 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:10:41.437 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:10:41.437 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:10:41.437 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-26T00:10:41.438 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-26T00:10:41.438 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.0 2024-03-26T00:10:41.438 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:10:41.438 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-26T00:10:41.439 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-26T00:10:41.440 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:10:41.440 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:10:41.440 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:10:41.471 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:10:41.472+0000 7f2a0db3e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:10:41.474 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:10:41.476+0000 7f2a0db3e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:10:41.476 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:10:41.476+0000 7f2a0db3e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:10:41.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-26T00:10:41.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:10:41.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-26T00:10:41.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:10:41.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:10:41.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:10:41.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:10:41.758 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:10:41.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:10:41.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:10:42.001 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:10:42.000+0000 7f2a0db3e7c0 -1 Falling back to public interface 2024-03-26T00:10:42.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:10:43.092 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:10:43.092 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:10:43.092 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:10:43.092 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:10:43.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:10:43.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:10:43.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:10:44.407 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:10:44.407 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:10:44.407 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:10:44.407 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:10:44.408 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:10:44.408 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:10:44.714 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:10:44.731 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:10:44.732+0000 7f2a0db3e7c0 -1 osd.0 44 log_to_monitors true 2024-03-26T00:10:45.715 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:10:45.715 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:10:45.715 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:10:45.715 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:10:45.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:10:45.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:10:46.204 INFO:tasks.workunit.client.0.smithi129.stderr:osd.0 up in weight 1 up_from 49 up_thru 49 down_at 45 last_clean_interval [5,44) [v2:127.0.0.1:6802/3527883874,v1:127.0.0.1:6803/3527883874] [v2:127.0.0.1:6804/3527883874,v1:127.0.0.1:6805/3527883874] exists,up 27293645-99f8-4138-92f4-30c701220322 2024-03-26T00:10:46.204 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:10:46.204 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:10:46.204 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:10:46.204 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:10:46.205 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:10:46.205 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:10:46.205 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:10:46.205 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:10:46.206 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:10:46.206 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:10:46.207 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:10:46.207 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:10:46.207 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:10:46.327 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:10:46.328 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:10:46.328 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:10:46.328 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:10:46.328 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:10:46.328 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:10:46.644 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:10:46.644 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:10:46.644 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-26T00:10:46.644 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:10:46.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:10:46.645 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:10:46.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397506 2024-03-26T00:10:46.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397506 2024-03-26T00:10:46.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-210453397506' 2024-03-26T00:10:46.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:10:46.789 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:10:46.939 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691846 2024-03-26T00:10:46.939 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691846 2024-03-26T00:10:46.940 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-210453397506 1-171798691846' 2024-03-26T00:10:46.940 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:10:46.940 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:10:47.082 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509461 2024-03-26T00:10:47.082 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509461 2024-03-26T00:10:47.082 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-210453397506 1-171798691846 2-64424509461' 2024-03-26T00:10:47.082 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:10:47.083 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-210453397506 2024-03-26T00:10:47.083 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:10:47.083 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:10:47.084 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-210453397506 2024-03-26T00:10:47.084 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:10:47.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397506 2024-03-26T00:10:47.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 210453397506' 2024-03-26T00:10:47.086 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 210453397506 2024-03-26T00:10:47.086 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:10:47.384 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 210453397506 2024-03-26T00:10:47.384 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:10:48.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:10:48.386 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:10:48.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397506 -lt 210453397506 2024-03-26T00:10:48.699 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:10:48.700 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691846 2024-03-26T00:10:48.700 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:10:48.702 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:10:48.703 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691846 2024-03-26T00:10:48.703 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:10:48.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691846 2024-03-26T00:10:48.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691846' 2024-03-26T00:10:48.705 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 171798691846 2024-03-26T00:10:48.705 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:10:49.044 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691846 -lt 171798691846 2024-03-26T00:10:49.044 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:10:49.045 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509461 2024-03-26T00:10:49.045 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:10:49.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:10:49.049 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509461 2024-03-26T00:10:49.049 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:10:49.050 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509461 2024-03-26T00:10:49.050 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-03-26T00:10:49.051 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 64424509461 2024-03-26T00:10:49.051 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:10:49.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509461 -lt 64424509461 2024-03-26T00:10:49.359 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:10:49.360 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:10:49.360 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:10:49.762 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:10:49.762 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:10:49.762 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:10:49.762 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:10:49.762 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:10:49.763 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:10:49.763 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:10:49.763 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:10:50.056 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:10:50.057 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:10:50.057 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:10:50.057 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:10:50.427 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:10:50.427 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:10:50.427 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:10:50.427 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-26T00:10:50.427 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ3 2024-03-26T00:10:50.427 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2024-03-26T00:10:50.427 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-26T00:10:50.427 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-26T00:10:50.427 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ3 2024-03-26T00:10:50.427 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-26T00:10:50.427 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-26T00:10:50.427 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-26T00:10:50.851 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-26T00:10:50.871 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-26T00:10:51.395 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-26T00:10:51.417 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-26T00:10:51.417 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-26T00:10:51.417 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:10:51.729 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 3 % 2 2024-03-26T00:10:51.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-03-26T00:10:51.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-26T00:10:51.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2024-03-26T00:10:51.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:10:51.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:10:51.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-26T00:10:51.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:10:51.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2024-03-26T00:10:51.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:10:51.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:10:51.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-26T00:10:51.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:10:51.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-26T00:10:51.733 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:10:51.733 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:10:51.733 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:10:51.733 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:10:51.733 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:10:53.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:10:53.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2024-03-26T00:10:53.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:10:53.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:10:53.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:10:53.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:10:53.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:10:53.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2024-03-26T00:10:54.258 INFO:tasks.workunit.client.0.smithi129.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2024-03-26T00:10:54.799 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-26T00:10:54.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:10:54.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:10:54.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-26T00:10:54.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:10:54.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:10:54.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:10:54.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:10:54.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:10:54.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:10:54.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:10:54.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:10:54.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:10:54.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:10:54.801 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:10:54.802 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:10:54.803 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:10:54.803 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:10:54.803 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:10:54.803 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:10:54.803 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:10:54.803 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:10:54.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:10:54.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:10:54.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:10:54.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:10:54.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:10:54.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:10:54.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:10:54.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:10:54.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:10:54.804 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:10:54.804 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:10:54.804 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-26T00:10:54.804 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.1 2024-03-26T00:10:54.804 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:10:54.804 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-26T00:10:54.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-26T00:10:54.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:10:54.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:10:54.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:10:54.837 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:10:54.836+0000 7f5c832dd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:10:54.840 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:10:54.840+0000 7f5c832dd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:10:54.841 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:10:54.840+0000 7f5c832dd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:10:55.134 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-26T00:10:55.134 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:10:55.134 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:10:55.134 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:10:55.134 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:10:55.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:10:55.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:10:55.135 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:10:55.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:10:55.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:10:55.385 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:10:55.384+0000 7f5c832dd7c0 -1 Falling back to public interface 2024-03-26T00:10:55.465 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:10:56.469 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:10:56.469 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:10:56.470 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:10:56.470 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:10:56.470 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:10:56.470 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:10:56.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:10:57.768 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:10:57.768 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:10:57.768 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:10:57.768 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:10:57.769 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:10:57.769 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:10:57.961 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:10:57.960+0000 7f5c832dd7c0 -1 osd.1 53 log_to_monitors true 2024-03-26T00:10:58.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:10:59.083 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:10:59.083 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:10:59.083 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:10:59.083 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:10:59.084 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:10:59.084 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:10:59.430 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1 up in weight 1 up_from 57 up_thru 57 down_at 54 last_clean_interval [40,53) [v2:127.0.0.1:6810/2457496232,v1:127.0.0.1:6811/2457496232] [v2:127.0.0.1:6812/2457496232,v1:127.0.0.1:6813/2457496232] exists,up 4d7ea6fb-1a4b-4612-ace7-7c4560a2fc00 2024-03-26T00:10:59.430 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:10:59.430 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:10:59.430 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:10:59.430 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:10:59.430 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:10:59.430 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:10:59.430 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:10:59.431 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:10:59.431 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:10:59.431 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:10:59.431 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:10:59.431 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:10:59.432 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:10:59.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:10:59.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:10:59.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:10:59.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:10:59.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:10:59.541 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:10:59.928 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:10:59.929 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:10:59.929 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-26T00:10:59.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:10:59.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:10:59.929 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:11:00.274 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397510 2024-03-26T00:11:00.274 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397510 2024-03-26T00:11:00.274 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-210453397510' 2024-03-26T00:11:00.274 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:11:00.275 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:11:00.424 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135874 2024-03-26T00:11:00.424 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135874 2024-03-26T00:11:00.424 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-210453397510 1-244813135874' 2024-03-26T00:11:00.424 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:11:00.424 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:11:00.594 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509465 2024-03-26T00:11:00.595 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509465 2024-03-26T00:11:00.595 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-210453397510 1-244813135874 2-64424509465' 2024-03-26T00:11:00.595 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:11:00.595 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-210453397510 2024-03-26T00:11:00.595 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:11:00.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:11:00.597 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-210453397510 2024-03-26T00:11:00.597 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:11:00.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397510 2024-03-26T00:11:00.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 210453397510' 2024-03-26T00:11:00.599 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 210453397510 2024-03-26T00:11:00.599 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:11:00.909 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397508 -lt 210453397510 2024-03-26T00:11:00.909 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:11:01.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:11:01.912 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:11:02.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397510 -lt 210453397510 2024-03-26T00:11:02.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:11:02.218 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-244813135874 2024-03-26T00:11:02.218 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:11:02.219 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:11:02.220 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-244813135874 2024-03-26T00:11:02.220 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:11:02.222 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135874 2024-03-26T00:11:02.222 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 244813135874' 2024-03-26T00:11:02.222 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 244813135874 2024-03-26T00:11:02.222 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:11:02.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135874 -lt 244813135874 2024-03-26T00:11:02.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:11:02.533 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509465 2024-03-26T00:11:02.533 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:11:02.535 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:11:02.536 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509465 2024-03-26T00:11:02.536 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:11:02.537 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509465 2024-03-26T00:11:02.538 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509465' 2024-03-26T00:11:02.538 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 64424509465 2024-03-26T00:11:02.538 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:11:02.876 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509465 -lt 64424509465 2024-03-26T00:11:02.877 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:11:02.877 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:11:02.878 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:11:03.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:11:03.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:11:03.264 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:11:03.264 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:11:03.264 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:11:03.264 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:11:03.264 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:11:03.264 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:11:03.600 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-26T00:11:03.600 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:11:03.601 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:11:03.601 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:11:03.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-03-26T00:11:03.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' -1 2024-03-26T00:11:03.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-26T00:11:03.953 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=5 2024-03-26T00:11:03.953 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-26T00:11:03.953 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-26T00:11:04.053 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-26T00:11:04.054 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:11:04.058 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:11:04.058 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:11:04.058 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:11:04.058 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:11:04.058 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:11:04.058 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:11:04.358 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-26T00:11:04.359 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:11:04.359 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:11:04.359 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:11:04.717 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-03-26T00:11:04.717 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' 5 2024-03-26T00:11:04.717 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-26T00:11:04.717 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-26T00:11:04.717 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-26T00:11:04.717 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-26T00:11:04.717 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-26T00:11:04.718 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-26T00:11:04.718 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-26T00:11:05.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-26T00:11:05.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-26T00:11:05.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-26T00:11:05.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-26T00:11:05.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-26T00:11:05.286 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-26T00:11:05.286 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:11:05.287 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:11:05.287 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:11:05.287 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:11:05.287 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:11:05.287 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:11:05.287 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:11:05.600 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-26T00:11:05.601 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:11:05.602 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:11:05.602 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:11:05.978 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-03-26T00:11:05.978 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' 5 2024-03-26T00:11:05.978 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-26T00:11:05.978 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-26T00:11:05.978 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-26T00:11:05.978 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-26T00:11:05.978 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-26T00:11:05.979 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-26T00:11:05.979 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-26T00:11:06.368 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-26T00:11:06.368 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-26T00:11:06.368 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-26T00:11:06.368 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-26T00:11:06.368 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-26T00:11:06.770 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-26T00:11:06.770 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:11:06.772 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:11:06.772 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:11:06.772 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:11:06.772 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:11:06.772 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:11:06.772 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:11:07.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:11:07.071 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:11:07.071 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:11:07.071 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:11:07.447 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:11:07.448 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:11:07.448 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:11:07.448 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-26T00:11:07.448 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ4 2024-03-26T00:11:07.448 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2024-03-26T00:11:07.448 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-26T00:11:07.448 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-26T00:11:07.448 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ4 2024-03-26T00:11:07.448 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-26T00:11:07.448 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-26T00:11:07.448 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-26T00:11:07.921 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-26T00:11:07.942 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-26T00:11:08.372 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-26T00:11:08.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-26T00:11:08.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-26T00:11:08.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:11:08.720 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 4 % 2 2024-03-26T00:11:08.722 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-03-26T00:11:08.722 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-26T00:11:08.722 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3638: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2024-03-26T00:11:09.030 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2024-03-26T00:11:09.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3642: corrupt_scrub_erasure: echo -n bad-val 2024-03-26T00:11:09.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3643: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-03-26T00:11:09.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:11:09.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:11:09.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-26T00:11:09.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:11:09.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-03-26T00:11:09.183 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:11:09.183 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:11:09.183 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-26T00:11:09.183 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:11:09.183 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-26T00:11:09.184 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:11:09.184 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:11:09.184 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:11:09.184 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:11:09.184 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:11:09.495 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:11:09.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-03-26T00:11:09.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:11:09.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:11:09.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:11:09.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:11:09.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:11:09.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-03-26T00:11:12.049 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-26T00:11:12.049 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:11:12.049 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:11:12.049 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-26T00:11:12.049 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:11:12.049 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:11:12.049 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:11:12.050 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:11:12.050 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:11:12.050 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:11:12.050 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-26T00:11:12.050 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-26T00:11:12.050 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:11:12.050 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:11:12.050 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:11:12.050 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:11:12.050 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:11:12.050 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:11:12.051 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:11:12.051 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:11:12.052 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:11:12.052 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:11:12.052 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:11:12.052 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:11:12.052 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:11:12.052 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:11:12.052 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:11:12.052 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:11:12.052 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:11:12.052 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:11:12.052 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:11:12.052 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:11:12.052 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-26T00:11:12.053 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-26T00:11:12.053 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.0 2024-03-26T00:11:12.053 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:11:12.053 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-26T00:11:12.054 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-26T00:11:12.055 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:11:12.055 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:11:12.055 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:11:12.086 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:11:12.088+0000 7f3061fe47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:11:12.089 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:11:12.088+0000 7f3061fe47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:11:12.090 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:11:12.092+0000 7f3061fe47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:11:12.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-26T00:11:12.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:11:12.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-26T00:11:12.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:11:12.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:11:12.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:11:12.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:11:12.371 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:11:12.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:11:12.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:11:12.628 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:11:12.628+0000 7f3061fe47c0 -1 Falling back to public interface 2024-03-26T00:11:12.687 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:11:13.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:11:13.690 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:11:13.690 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:11:13.690 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:11:13.690 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:11:13.690 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:11:14.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:11:15.012 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:11:15.012 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:11:15.012 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:11:15.012 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:11:15.012 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:11:15.013 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:11:15.206 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:11:15.203+0000 7f3061fe47c0 -1 osd.0 61 log_to_monitors true 2024-03-26T00:11:15.338 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:11:16.339 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:11:16.340 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:11:16.340 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:11:16.340 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:11:16.340 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:11:16.340 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:11:16.782 INFO:tasks.workunit.client.0.smithi129.stderr:osd.0 up in weight 1 up_from 66 up_thru 66 down_at 62 last_clean_interval [49,61) [v2:127.0.0.1:6802/4188046581,v1:127.0.0.1:6803/4188046581] [v2:127.0.0.1:6804/4188046581,v1:127.0.0.1:6805/4188046581] exists,up 27293645-99f8-4138-92f4-30c701220322 2024-03-26T00:11:16.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:11:16.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:11:16.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:11:16.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:11:16.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:11:16.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:11:16.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:11:16.783 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:11:16.783 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:11:16.784 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:11:16.784 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:11:16.784 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:11:16.784 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:11:16.894 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:11:16.894 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:11:16.894 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:11:16.894 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:11:16.894 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:11:16.894 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:11:17.268 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:11:17.269 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:11:17.269 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-26T00:11:17.269 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:11:17.269 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:11:17.269 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:11:17.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841538 2024-03-26T00:11:17.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841538 2024-03-26T00:11:17.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841538' 2024-03-26T00:11:17.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:11:17.420 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:11:17.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135878 2024-03-26T00:11:17.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135878 2024-03-26T00:11:17.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841538 1-244813135878' 2024-03-26T00:11:17.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:11:17.565 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:11:17.705 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509469 2024-03-26T00:11:17.705 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509469 2024-03-26T00:11:17.705 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841538 1-244813135878 2-64424509469' 2024-03-26T00:11:17.705 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:11:17.706 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:11:17.706 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-283467841538 2024-03-26T00:11:17.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:11:17.707 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-283467841538 2024-03-26T00:11:17.707 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:11:17.709 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841538 2024-03-26T00:11:17.709 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 283467841538' 2024-03-26T00:11:17.709 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 283467841538 2024-03-26T00:11:17.709 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:11:18.009 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841538 -lt 283467841538 2024-03-26T00:11:18.009 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:11:18.009 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-244813135878 2024-03-26T00:11:18.009 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:11:18.011 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:11:18.012 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-244813135878 2024-03-26T00:11:18.012 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:11:18.013 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135878 2024-03-26T00:11:18.013 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 244813135878' 2024-03-26T00:11:18.013 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 244813135878 2024-03-26T00:11:18.013 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:11:18.335 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135878 -lt 244813135878 2024-03-26T00:11:18.335 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:11:18.336 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509469 2024-03-26T00:11:18.336 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:11:18.338 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:11:18.338 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509469 2024-03-26T00:11:18.339 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:11:18.340 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509469 2024-03-26T00:11:18.340 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509469' 2024-03-26T00:11:18.341 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 64424509469 2024-03-26T00:11:18.341 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:11:18.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509469 -lt 64424509469 2024-03-26T00:11:18.645 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:11:18.646 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:11:18.646 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:11:19.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:11:19.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:11:19.034 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:11:19.034 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:11:19.034 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:11:19.034 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:11:19.034 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:11:19.034 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:11:19.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:11:19.509 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:11:19.510 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:11:19.510 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:11:19.882 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:11:19.883 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:11:19.883 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:11:19.883 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3644: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-03-26T00:11:19.883 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:11:19.883 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:11:19.883 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-03-26T00:11:19.883 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:11:19.883 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-03-26T00:11:19.883 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:11:19.883 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:11:19.883 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-03-26T00:11:19.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:11:19.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-26T00:11:19.884 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:11:19.884 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:11:19.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:11:19.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:11:19.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:11:20.193 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:11:20.193 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-03-26T00:11:20.193 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:11:20.193 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:11:20.193 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-03-26T00:11:20.193 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:11:20.193 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-03-26T00:11:20.193 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2024-03-26T00:11:22.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-03-26T00:11:22.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:11:22.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:11:22.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-26T00:11:22.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:11:22.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-26T00:11:22.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:11:22.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:11:22.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:11:22.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:11:22.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-26T00:11:22.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-26T00:11:22.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:11:22.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:11:22.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:11:22.779 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:11:22.779 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:11:22.779 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:11:22.781 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:11:22.781 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:11:22.781 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:11:22.781 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:11:22.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:11:22.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:11:22.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:11:22.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:11:22.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:11:22.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:11:22.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:11:22.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:11:22.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:11:22.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:11:22.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-26T00:11:22.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-26T00:11:22.783 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.2 2024-03-26T00:11:22.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:11:22.783 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-26T00:11:22.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-26T00:11:22.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:11:22.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:11:22.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:11:22.815 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:11:22.815+0000 7f65c8aba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:11:22.818 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:11:22.819+0000 7f65c8aba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:11:22.820 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:11:22.819+0000 7f65c8aba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:11:23.105 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-26T00:11:23.106 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:11:23.106 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-26T00:11:23.106 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:11:23.106 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:11:23.106 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:11:23.106 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:11:23.106 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:11:23.107 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:11:23.107 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:11:23.353 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:11:23.351+0000 7f65c8aba7c0 -1 Falling back to public interface 2024-03-26T00:11:23.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:11:24.436 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:11:24.436 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:11:24.436 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:11:24.436 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:11:24.437 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:11:24.437 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:11:24.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:11:25.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:11:25.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:11:25.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:11:25.757 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:11:25.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:11:25.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:11:26.042 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:11:26.039+0000 7f65c8aba7c0 -1 osd.2 68 log_to_monitors true 2024-03-26T00:11:26.079 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:11:27.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:11:27.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:11:27.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:11:27.081 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:11:27.082 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:11:27.082 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:11:27.428 INFO:tasks.workunit.client.0.smithi129.stderr:osd.2 up in weight 1 up_from 73 up_thru 73 down_at 69 last_clean_interval [15,68) [v2:127.0.0.1:6818/3833207422,v1:127.0.0.1:6819/3833207422] [v2:127.0.0.1:6820/3833207422,v1:127.0.0.1:6821/3833207422] exists,up 8d661fa8-dc51-4e46-8532-fc7445949179 2024-03-26T00:11:27.429 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:11:27.429 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:11:27.429 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:11:27.429 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:11:27.429 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:11:27.429 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:11:27.429 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:11:27.430 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:11:27.430 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:11:27.430 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:11:27.430 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:11:27.430 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:11:27.430 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:11:27.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:11:27.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:11:27.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:11:27.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:11:27.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:11:27.550 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:11:27.937 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:11:27.938 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:11:27.938 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-26T00:11:27.938 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:11:27.938 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:11:27.938 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:11:28.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841541 2024-03-26T00:11:28.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841541 2024-03-26T00:11:28.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841541' 2024-03-26T00:11:28.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:11:28.177 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:11:28.320 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135882 2024-03-26T00:11:28.320 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135882 2024-03-26T00:11:28.320 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841541 1-244813135882' 2024-03-26T00:11:28.321 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:11:28.321 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:11:28.466 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612610 2024-03-26T00:11:28.466 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612610 2024-03-26T00:11:28.466 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841541 1-244813135882 2-313532612610' 2024-03-26T00:11:28.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:11:28.467 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-283467841541 2024-03-26T00:11:28.467 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:11:28.468 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:11:28.469 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:11:28.469 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-283467841541 2024-03-26T00:11:28.470 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841541 2024-03-26T00:11:28.470 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 283467841541' 2024-03-26T00:11:28.470 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 283467841541 2024-03-26T00:11:28.470 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:11:28.765 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841540 -lt 283467841541 2024-03-26T00:11:28.765 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:11:29.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:11:29.768 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:11:30.079 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841541 -lt 283467841541 2024-03-26T00:11:30.079 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:11:30.080 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-244813135882 2024-03-26T00:11:30.080 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:11:30.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:11:30.082 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-244813135882 2024-03-26T00:11:30.082 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:11:30.083 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135882 2024-03-26T00:11:30.083 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 244813135882' 2024-03-26T00:11:30.083 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 244813135882 2024-03-26T00:11:30.084 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:11:30.445 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135882 -lt 244813135882 2024-03-26T00:11:30.445 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:11:30.446 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-313532612610 2024-03-26T00:11:30.446 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:11:30.447 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:11:30.449 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-313532612610 2024-03-26T00:11:30.449 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:11:30.450 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612610 2024-03-26T00:11:30.450 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 313532612610' 2024-03-26T00:11:30.450 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 313532612610 2024-03-26T00:11:30.451 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:11:30.788 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612610 -lt 313532612610 2024-03-26T00:11:30.789 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:11:30.789 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:11:30.790 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:11:31.196 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:11:31.196 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:11:31.197 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:11:31.197 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:11:31.197 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:11:31.197 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:11:31.198 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:11:31.198 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:11:31.537 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:11:31.539 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:11:31.539 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:11:31.539 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:11:31.902 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:11:31.902 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:11:31.902 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:11:31.902 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3645: corrupt_scrub_erasure: echo -n val3-EOBJ4 2024-03-26T00:11:31.902 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3646: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-03-26T00:11:31.902 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:11:31.902 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:11:31.903 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-03-26T00:11:31.903 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:11:31.903 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-03-26T00:11:31.903 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:11:31.903 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:11:31.903 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-03-26T00:11:31.903 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:11:31.903 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-26T00:11:31.903 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:11:31.903 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:11:31.903 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:11:31.903 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:11:31.903 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:11:32.210 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:11:32.210 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-03-26T00:11:32.210 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:11:32.210 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:11:32.210 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-03-26T00:11:32.211 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:11:32.211 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-03-26T00:11:32.211 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-03-26T00:11:34.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-03-26T00:11:34.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:11:34.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:11:34.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-26T00:11:34.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:11:34.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-26T00:11:34.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:11:34.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:11:34.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:11:34.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:11:34.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-26T00:11:34.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-26T00:11:34.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:11:34.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:11:34.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:11:34.698 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:11:34.698 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:11:34.698 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:11:34.699 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:11:34.699 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:11:34.699 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:11:34.699 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:11:34.699 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:11:34.699 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:11:34.699 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:11:34.699 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:11:34.699 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:11:34.699 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:11:34.699 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:11:34.699 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:11:34.699 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:11:34.699 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:11:34.700 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-26T00:11:34.700 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-26T00:11:34.700 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.2 2024-03-26T00:11:34.700 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:11:34.700 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-26T00:11:34.700 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-26T00:11:34.700 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:11:34.701 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:11:34.701 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:11:34.729 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:11:34.727+0000 7fe45f5237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:11:34.732 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:11:34.731+0000 7fe45f5237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:11:34.733 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:11:34.731+0000 7fe45f5237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:11:35.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-26T00:11:35.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:11:35.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-26T00:11:35.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:11:35.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:11:35.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:11:35.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:11:35.015 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:11:35.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:11:35.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:11:35.277 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:11:35.275+0000 7fe45f5237c0 -1 Falling back to public interface 2024-03-26T00:11:35.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:11:36.354 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:11:36.355 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:11:36.355 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:11:36.355 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:11:36.355 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:11:36.355 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:11:36.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:11:37.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:11:37.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:11:37.666 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:11:37.666 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:11:37.666 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:11:37.666 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:11:37.975 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:11:38.032 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:11:38.031+0000 7fe45f5237c0 -1 osd.2 75 log_to_monitors true 2024-03-26T00:11:38.977 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:11:38.977 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:11:38.977 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:11:38.977 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:11:38.977 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:11:38.978 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:11:39.404 INFO:tasks.workunit.client.0.smithi129.stderr:osd.2 up in weight 1 up_from 80 up_thru 80 down_at 76 last_clean_interval [73,75) [v2:127.0.0.1:6818/3596175077,v1:127.0.0.1:6819/3596175077] [v2:127.0.0.1:6820/3596175077,v1:127.0.0.1:6821/3596175077] exists,up 8d661fa8-dc51-4e46-8532-fc7445949179 2024-03-26T00:11:39.404 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:11:39.404 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:11:39.404 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:11:39.404 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:11:39.404 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:11:39.404 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:11:39.404 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:11:39.405 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:11:39.405 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:11:39.405 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:11:39.406 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:11:39.406 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:11:39.406 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:11:39.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:11:39.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:11:39.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:11:39.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:11:39.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:11:39.513 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:11:39.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:11:39.892 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:11:39.892 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-26T00:11:39.893 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:11:39.893 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:11:39.893 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:11:40.245 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841544 2024-03-26T00:11:40.245 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841544 2024-03-26T00:11:40.245 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841544' 2024-03-26T00:11:40.245 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:11:40.245 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:11:40.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135885 2024-03-26T00:11:40.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135885 2024-03-26T00:11:40.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841544 1-244813135885' 2024-03-26T00:11:40.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:11:40.416 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:11:40.556 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383682 2024-03-26T00:11:40.557 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383682 2024-03-26T00:11:40.557 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841544 1-244813135885 2-343597383682' 2024-03-26T00:11:40.557 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:11:40.557 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-283467841544 2024-03-26T00:11:40.557 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:11:40.558 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:11:40.559 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-283467841544 2024-03-26T00:11:40.559 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:11:40.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841544 2024-03-26T00:11:40.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 283467841544' 2024-03-26T00:11:40.560 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 283467841544 2024-03-26T00:11:40.560 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:11:40.855 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841543 -lt 283467841544 2024-03-26T00:11:40.855 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:11:41.857 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:11:41.858 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:11:42.163 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841545 -lt 283467841544 2024-03-26T00:11:42.163 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:11:42.164 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-244813135885 2024-03-26T00:11:42.164 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:11:42.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:11:42.166 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-244813135885 2024-03-26T00:11:42.167 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:11:42.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135885 2024-03-26T00:11:42.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 244813135885' 2024-03-26T00:11:42.169 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 244813135885 2024-03-26T00:11:42.170 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:11:42.502 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135885 -lt 244813135885 2024-03-26T00:11:42.503 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:11:42.503 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383682 2024-03-26T00:11:42.503 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:11:42.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:11:42.505 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383682 2024-03-26T00:11:42.506 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:11:42.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383682 2024-03-26T00:11:42.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383682' 2024-03-26T00:11:42.507 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 343597383682 2024-03-26T00:11:42.508 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:11:42.847 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383682 -lt 343597383682 2024-03-26T00:11:42.848 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:11:42.848 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:11:42.848 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:11:43.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:11:43.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:11:43.244 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:11:43.244 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:11:43.244 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:11:43.244 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:11:43.245 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:11:43.245 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:11:43.583 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:11:43.583 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:11:43.583 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:11:43.583 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:11:43.935 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:11:43.935 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:11:43.935 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:11:43.935 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3647: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-03-26T00:11:43.937 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-26T00:11:43.937 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ5 2024-03-26T00:11:43.938 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2024-03-26T00:11:43.938 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-26T00:11:43.938 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-26T00:11:43.938 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ5 2024-03-26T00:11:43.938 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-26T00:11:43.938 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-26T00:11:43.938 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-26T00:11:44.378 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-26T00:11:44.398 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-26T00:11:44.871 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-26T00:11:44.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-26T00:11:44.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-26T00:11:44.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:11:45.162 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 5 % 2 2024-03-26T00:11:45.163 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-03-26T00:11:45.163 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-26T00:11:45.163 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3652: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2024-03-26T00:11:45.164 INFO:tasks.workunit.client.0.smithi129.stderr:2+0 records in 2024-03-26T00:11:45.164 INFO:tasks.workunit.client.0.smithi129.stderr:2+0 records out 2024-03-26T00:11:45.165 INFO:tasks.workunit.client.0.smithi129.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000209486 s, 19.6 MB/s 2024-03-26T00:11:45.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3653: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-26T00:11:45.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:11:45.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:11:45.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-26T00:11:45.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:11:45.167 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-26T00:11:45.167 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:11:45.167 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:11:45.167 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-26T00:11:45.167 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:11:45.167 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-26T00:11:45.168 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:11:45.168 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:11:45.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:11:45.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:11:45.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:11:46.480 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:11:46.480 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-26T00:11:46.480 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:11:46.480 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:11:46.480 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:11:46.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:11:46.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:11:46.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-26T00:11:48.214 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-26T00:11:48.214 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:11:48.215 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:11:48.215 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-26T00:11:48.215 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:11:48.215 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:11:48.215 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:11:48.215 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:11:48.215 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:11:48.215 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:11:48.215 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:11:48.215 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:11:48.215 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:11:48.215 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:11:48.215 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:11:48.215 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:11:48.216 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:11:48.216 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:11:48.217 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:11:48.217 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:11:48.217 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:11:48.217 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:11:48.218 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:11:48.218 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:11:48.218 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:11:48.218 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:11:48.218 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:11:48.218 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:11:48.218 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:11:48.218 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:11:48.218 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:11:48.218 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:11:48.218 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:11:48.219 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-26T00:11:48.219 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.1 2024-03-26T00:11:48.219 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:11:48.219 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-26T00:11:48.220 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-26T00:11:48.220 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:11:48.220 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:11:48.221 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:11:48.252 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:11:48.251+0000 7fc10111f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:11:48.255 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:11:48.255+0000 7fc10111f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:11:48.257 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:11:48.255+0000 7fc10111f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:11:48.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-26T00:11:48.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:11:48.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:11:48.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:11:48.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:11:48.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:11:48.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:11:48.566 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:11:48.566 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:11:48.566 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:11:48.801 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:11:48.799+0000 7fc10111f7c0 -1 Falling back to public interface 2024-03-26T00:11:48.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:11:49.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:11:49.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:11:49.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:11:49.917 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:11:49.918 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:11:49.918 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:11:50.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:11:51.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:11:51.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:11:51.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:11:51.280 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:11:51.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:11:51.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:11:51.463 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:11:51.463+0000 7fc10111f7c0 -1 osd.1 84 log_to_monitors true 2024-03-26T00:11:51.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:11:52.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:11:52.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:11:52.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:11:52.609 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:11:52.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:11:52.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:11:52.949 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1 up in weight 1 up_from 89 up_thru 80 down_at 85 last_clean_interval [57,84) [v2:127.0.0.1:6810/2474525786,v1:127.0.0.1:6811/2474525786] [v2:127.0.0.1:6812/2474525786,v1:127.0.0.1:6813/2474525786] exists,up 4d7ea6fb-1a4b-4612-ace7-7c4560a2fc00 2024-03-26T00:11:52.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:11:52.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:11:52.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:11:52.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:11:52.950 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:11:52.950 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:11:52.950 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:11:52.950 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:11:52.951 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:11:52.951 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:11:52.951 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:11:52.951 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:11:52.951 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:11:53.056 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:11:53.056 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:11:53.056 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:11:53.056 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:11:53.057 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:11:53.057 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:11:53.465 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:11:53.465 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:11:53.465 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-26T00:11:53.466 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:11:53.466 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:11:53.466 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:11:53.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841548 2024-03-26T00:11:53.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841548 2024-03-26T00:11:53.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841548' 2024-03-26T00:11:53.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:11:53.698 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:11:53.844 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=382252089346 2024-03-26T00:11:53.844 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 382252089346 2024-03-26T00:11:53.844 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841548 1-382252089346' 2024-03-26T00:11:53.844 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:11:53.844 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:11:54.011 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383686 2024-03-26T00:11:54.011 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383686 2024-03-26T00:11:54.011 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841548 1-382252089346 2-343597383686' 2024-03-26T00:11:54.011 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:11:54.011 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-283467841548 2024-03-26T00:11:54.011 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:11:54.012 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:11:54.013 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-283467841548 2024-03-26T00:11:54.013 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:11:54.014 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841548 2024-03-26T00:11:54.014 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 283467841548' 2024-03-26T00:11:54.014 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 283467841548 2024-03-26T00:11:54.015 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:11:54.318 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841548 -lt 283467841548 2024-03-26T00:11:54.318 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:11:54.319 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:11:54.319 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-382252089346 2024-03-26T00:11:54.320 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:11:54.321 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-382252089346 2024-03-26T00:11:54.321 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:11:54.323 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=382252089346 2024-03-26T00:11:54.323 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 382252089346' 2024-03-26T00:11:54.323 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 382252089346 2024-03-26T00:11:54.323 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:11:54.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 382252089346 2024-03-26T00:11:54.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:11:55.636 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:11:55.637 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:11:55.982 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 382252089346 -lt 382252089346 2024-03-26T00:11:55.983 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:11:55.983 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383686 2024-03-26T00:11:55.983 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:11:55.985 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:11:55.986 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383686 2024-03-26T00:11:55.986 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:11:55.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383686 2024-03-26T00:11:55.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383686' 2024-03-26T00:11:55.988 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 343597383686 2024-03-26T00:11:55.988 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:11:56.314 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383686 -lt 343597383686 2024-03-26T00:11:56.314 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:11:56.315 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:11:56.315 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:11:56.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:11:56.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:11:56.717 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:11:56.717 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:11:56.717 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:11:56.717 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:11:56.717 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:11:56.717 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:11:57.080 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:11:57.081 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:11:57.081 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:11:57.081 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:11:57.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:11:57.462 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:11:57.462 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:11:57.462 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-26T00:11:57.462 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ6 2024-03-26T00:11:57.462 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2024-03-26T00:11:57.462 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-26T00:11:57.462 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-26T00:11:57.462 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ6 2024-03-26T00:11:57.462 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-26T00:11:57.462 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-26T00:11:57.462 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-26T00:11:57.926 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-26T00:11:57.947 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-26T00:11:58.427 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-26T00:11:58.449 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-26T00:11:58.449 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-26T00:11:58.449 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:11:58.726 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 6 % 2 2024-03-26T00:11:58.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-03-26T00:11:58.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-26T00:11:58.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3657: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-03-26T00:11:58.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:11:58.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:11:58.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-26T00:11:58.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:11:58.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-03-26T00:11:58.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:11:58.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:11:58.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-26T00:11:58.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:11:58.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-26T00:11:58.732 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:11:58.732 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:11:58.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:11:58.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:11:58.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:11:59.054 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:11:59.054 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-03-26T00:11:59.054 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:11:59.054 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:11:59.054 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:11:59.054 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:11:59.055 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:11:59.055 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2024-03-26T00:12:01.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-26T00:12:01.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:12:01.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:12:01.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-26T00:12:01.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:12:01.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:12:01.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:12:01.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:12:01.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:12:01.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:12:01.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-26T00:12:01.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-26T00:12:01.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:12:01.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:12:01.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:12:01.562 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:12:01.562 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:12:01.562 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:12:01.564 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:12:01.564 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:12:01.564 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:12:01.564 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:12:01.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:12:01.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:12:01.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:12:01.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:12:01.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:12:01.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:12:01.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:12:01.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:12:01.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:12:01.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:12:01.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-26T00:12:01.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-26T00:12:01.565 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.0 2024-03-26T00:12:01.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:12:01.566 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-26T00:12:01.566 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-26T00:12:01.566 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:12:01.566 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:12:01.566 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:12:01.598 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:12:01.594+0000 7fd150ee47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:12:01.601 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:12:01.598+0000 7fd150ee47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:12:01.602 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:12:01.598+0000 7fd150ee47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:12:01.895 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-26T00:12:01.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:12:01.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-26T00:12:01.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:12:01.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:12:01.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:12:01.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:12:01.896 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:12:01.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:12:01.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:12:02.137 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:12:02.134+0000 7fd150ee47c0 -1 Falling back to public interface 2024-03-26T00:12:02.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:12:03.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:12:03.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:12:03.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:12:03.188 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:12:03.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:12:03.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:12:03.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:12:04.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:12:04.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:12:04.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:12:04.509 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:12:04.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:12:04.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:12:04.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:12:04.900 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:12:04.898+0000 7fd150ee47c0 -1 osd.0 93 log_to_monitors true 2024-03-26T00:12:05.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:12:05.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:12:05.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:12:05.832 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:12:05.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:12:05.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:12:06.293 INFO:tasks.workunit.client.0.smithi129.stderr:osd.0 up in weight 1 up_from 98 up_thru 98 down_at 94 last_clean_interval [66,93) [v2:127.0.0.1:6802/1800929122,v1:127.0.0.1:6803/1800929122] [v2:127.0.0.1:6804/1800929122,v1:127.0.0.1:6805/1800929122] exists,up 27293645-99f8-4138-92f4-30c701220322 2024-03-26T00:12:06.293 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:12:06.293 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:12:06.293 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:12:06.293 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:12:06.293 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:12:06.293 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:12:06.293 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:12:06.294 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:12:06.295 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:12:06.295 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:12:06.295 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:12:06.295 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:12:06.295 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:12:06.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:12:06.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:12:06.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:12:06.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:12:06.416 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:12:06.416 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:12:06.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:12:06.927 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:12:06.927 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-26T00:12:06.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:12:06.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:12:06.928 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:12:07.131 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=420906795010 2024-03-26T00:12:07.131 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 420906795010 2024-03-26T00:12:07.131 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795010' 2024-03-26T00:12:07.131 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:12:07.131 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:12:07.275 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=382252089350 2024-03-26T00:12:07.275 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 382252089350 2024-03-26T00:12:07.275 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795010 1-382252089350' 2024-03-26T00:12:07.275 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:12:07.275 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:12:07.416 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383689 2024-03-26T00:12:07.417 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383689 2024-03-26T00:12:07.417 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795010 1-382252089350 2-343597383689' 2024-03-26T00:12:07.417 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:12:07.417 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-420906795010 2024-03-26T00:12:07.417 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:12:07.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:12:07.419 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-420906795010 2024-03-26T00:12:07.419 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:12:07.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=420906795010 2024-03-26T00:12:07.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 420906795010' 2024-03-26T00:12:07.420 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 420906795010 2024-03-26T00:12:07.420 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:12:07.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 420906795010 2024-03-26T00:12:07.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:12:08.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:12:08.706 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:12:09.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 420906795010 -lt 420906795010 2024-03-26T00:12:09.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:12:09.018 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-382252089350 2024-03-26T00:12:09.018 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:12:09.019 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:12:09.020 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-382252089350 2024-03-26T00:12:09.020 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:12:09.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=382252089350 2024-03-26T00:12:09.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 382252089350' 2024-03-26T00:12:09.021 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 382252089350 2024-03-26T00:12:09.022 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:12:09.375 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 382252089350 -lt 382252089350 2024-03-26T00:12:09.375 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:12:09.375 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383689 2024-03-26T00:12:09.376 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:12:09.377 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:12:09.378 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383689 2024-03-26T00:12:09.378 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:12:09.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383689 2024-03-26T00:12:09.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383689' 2024-03-26T00:12:09.379 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 343597383689 2024-03-26T00:12:09.380 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:12:09.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383689 -lt 343597383689 2024-03-26T00:12:09.705 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:12:09.705 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:12:09.705 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:12:10.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:12:10.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:12:10.123 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:12:10.123 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:12:10.123 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:12:10.123 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:12:10.124 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:12:10.124 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:12:10.474 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-26T00:12:10.475 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:12:10.475 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:12:10.475 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:12:10.849 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-26T00:12:10.849 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-26T00:12:10.849 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-26T00:12:10.849 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-26T00:12:10.849 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-26T00:12:10.849 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-26T00:12:10.950 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-26T00:12:10.950 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:12:10.951 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:12:10.951 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:12:10.952 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:12:10.952 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:12:10.952 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:12:10.952 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:12:11.250 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-26T00:12:11.250 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:12:11.251 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:12:11.251 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:12:11.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-26T00:12:11.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-26T00:12:11.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-26T00:12:11.614 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-26T00:12:11.614 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-26T00:12:11.614 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-26T00:12:11.614 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-26T00:12:11.614 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-26T00:12:11.614 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-26T00:12:12.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-26T00:12:12.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-26T00:12:12.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-26T00:12:12.028 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-26T00:12:12.028 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-26T00:12:12.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-26T00:12:12.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:12:12.230 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:12:12.230 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:12:12.231 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:12:12.231 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:12:12.231 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:12:12.231 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:12:12.545 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-26T00:12:12.546 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:12:12.546 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:12:12.547 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:12:12.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-26T00:12:12.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-26T00:12:12.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-26T00:12:12.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-26T00:12:12.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-26T00:12:12.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-26T00:12:12.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-26T00:12:12.923 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-26T00:12:12.923 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-26T00:12:13.300 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-26T00:12:13.300 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-26T00:12:13.300 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-26T00:12:13.300 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-26T00:12:13.300 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-26T00:12:13.701 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-26T00:12:13.702 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:12:13.703 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:12:13.703 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:12:13.703 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:12:13.703 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:12:13.704 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:12:13.704 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:12:14.013 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:12:14.013 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:12:14.014 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:12:14.014 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:12:14.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:12:14.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:12:14.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:12:14.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3658: corrupt_scrub_erasure: echo -n bad-val 2024-03-26T00:12:14.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3659: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-03-26T00:12:14.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:12:14.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:12:14.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-26T00:12:14.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:12:14.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-03-26T00:12:14.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:12:14.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:12:14.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-26T00:12:14.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:12:14.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-26T00:12:14.388 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:12:14.388 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:12:14.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:12:14.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:12:14.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:12:14.696 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:12:14.696 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-03-26T00:12:14.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:12:14.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:12:14.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:12:14.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:12:14.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:12:14.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-03-26T00:12:16.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-26T00:12:16.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:12:16.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:12:16.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-26T00:12:16.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:12:16.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:12:16.801 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:12:16.801 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:12:16.801 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:12:16.801 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:12:16.801 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:12:16.801 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:12:16.801 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:12:16.801 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:12:16.801 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:12:16.801 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:12:16.801 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:12:16.801 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:12:16.805 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:12:16.805 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:12:16.805 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:12:16.806 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:12:16.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:12:16.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:12:16.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:12:16.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:12:16.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:12:16.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:12:16.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:12:16.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:12:16.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:12:16.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:12:16.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:12:16.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-26T00:12:16.806 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.1 2024-03-26T00:12:16.808 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:12:16.808 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-26T00:12:16.808 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-26T00:12:16.808 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:12:16.808 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:12:16.808 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:12:16.837 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:12:16.834+0000 7fe578fac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:12:16.840 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:12:16.838+0000 7fe578fac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:12:16.841 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:12:16.838+0000 7fe578fac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:12:17.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-26T00:12:17.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:12:17.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:12:17.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:12:17.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:12:17.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:12:17.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:12:17.132 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:12:17.133 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:12:17.133 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:12:17.376 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:12:17.374+0000 7fe578fac7c0 -1 Falling back to public interface 2024-03-26T00:12:17.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:12:18.516 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:12:18.516 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:12:18.516 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:12:18.516 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:12:18.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:12:18.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:12:18.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:12:19.816 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:12:19.816 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:12:19.816 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:12:19.816 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:12:19.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:12:19.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:12:20.064 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:12:20.062+0000 7fe578fac7c0 -1 osd.1 100 log_to_monitors true 2024-03-26T00:12:20.141 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:12:21.143 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:12:21.144 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:12:21.144 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:12:21.144 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:12:21.144 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:12:21.144 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:12:21.698 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1 up in weight 1 up_from 104 up_thru 104 down_at 101 last_clean_interval [89,100) [v2:127.0.0.1:6810/3953787274,v1:127.0.0.1:6811/3953787274] [v2:127.0.0.1:6812/3953787274,v1:127.0.0.1:6813/3953787274] exists,up 4d7ea6fb-1a4b-4612-ace7-7c4560a2fc00 2024-03-26T00:12:21.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:12:21.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:12:21.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:12:21.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:12:21.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:12:21.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:12:21.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:12:21.699 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:12:21.699 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:12:21.699 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:12:21.700 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:12:21.700 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:12:21.700 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:12:21.818 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:12:21.819 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:12:21.819 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:12:21.819 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:12:21.819 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:12:21.819 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:12:22.191 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:12:22.191 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:12:22.191 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-26T00:12:22.191 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:12:22.191 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:12:22.191 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:12:22.336 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=420906795014 2024-03-26T00:12:22.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 420906795014 2024-03-26T00:12:22.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795014' 2024-03-26T00:12:22.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:12:22.337 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:12:22.476 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=446676598786 2024-03-26T00:12:22.476 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 446676598786 2024-03-26T00:12:22.476 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795014 1-446676598786' 2024-03-26T00:12:22.476 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:12:22.476 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:12:22.606 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383693 2024-03-26T00:12:22.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383693 2024-03-26T00:12:22.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795014 1-446676598786 2-343597383693' 2024-03-26T00:12:22.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:12:22.607 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-420906795014 2024-03-26T00:12:22.607 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:12:22.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:12:22.609 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-420906795014 2024-03-26T00:12:22.609 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:12:22.610 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=420906795014 2024-03-26T00:12:22.610 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 420906795014' 2024-03-26T00:12:22.610 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 420906795014 2024-03-26T00:12:22.610 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:12:22.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 420906795013 -lt 420906795014 2024-03-26T00:12:22.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:12:23.900 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:12:23.901 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:12:24.211 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 420906795014 -lt 420906795014 2024-03-26T00:12:24.211 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:12:24.213 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-446676598786 2024-03-26T00:12:24.213 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:12:24.214 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:12:24.215 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-446676598786 2024-03-26T00:12:24.215 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:12:24.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=446676598786 2024-03-26T00:12:24.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 446676598786' 2024-03-26T00:12:24.217 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 446676598786 2024-03-26T00:12:24.218 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:12:24.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 446676598786 -lt 446676598786 2024-03-26T00:12:24.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:12:24.578 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383693 2024-03-26T00:12:24.578 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:12:24.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:12:24.580 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383693 2024-03-26T00:12:24.580 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:12:24.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383693 2024-03-26T00:12:24.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383693' 2024-03-26T00:12:24.581 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 343597383693 2024-03-26T00:12:24.581 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:12:24.876 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383694 -lt 343597383693 2024-03-26T00:12:24.876 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:12:24.877 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:12:24.877 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:12:25.250 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:12:25.250 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:12:25.251 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:12:25.251 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:12:25.251 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:12:25.251 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:12:25.251 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:12:25.252 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:12:25.570 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:12:25.570 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:12:25.571 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:12:25.571 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:12:25.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:12:25.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:12:25.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:12:25.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-26T00:12:25.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ7 2024-03-26T00:12:25.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2024-03-26T00:12:25.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-26T00:12:25.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-26T00:12:25.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ7 2024-03-26T00:12:25.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-26T00:12:25.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-26T00:12:25.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-26T00:12:26.338 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-26T00:12:26.356 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-26T00:12:26.924 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-26T00:12:26.946 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-26T00:12:26.947 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-26T00:12:26.947 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:12:27.105 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 7 % 2 2024-03-26T00:12:27.107 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-03-26T00:12:27.107 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-26T00:12:27.107 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3663: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-03-26T00:12:27.107 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3664: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-03-26T00:12:27.107 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3665: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2024-03-26T00:12:27.323 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3668: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-03-26T00:12:27.323 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:12:27.323 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:12:27.323 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-26T00:12:27.323 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:12:27.323 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-03-26T00:12:27.323 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:12:27.323 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:12:27.323 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-26T00:12:27.323 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:12:27.323 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-26T00:12:27.323 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:12:27.323 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:12:27.324 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:12:27.324 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:12:27.324 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:12:27.635 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:12:27.635 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-03-26T00:12:27.635 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:12:27.635 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:12:27.635 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:12:27.635 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:12:27.635 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:12:27.635 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2024-03-26T00:12:29.720 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-26T00:12:29.721 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:12:29.721 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:12:29.721 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-26T00:12:29.721 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:12:29.721 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:12:29.721 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:12:29.721 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:12:29.721 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:12:29.721 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:12:29.721 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-26T00:12:29.721 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-26T00:12:29.722 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:12:29.722 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:12:29.722 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:12:29.723 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:12:29.724 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:12:29.724 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:12:29.724 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:12:29.724 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:12:29.724 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:12:29.724 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:12:29.724 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:12:29.724 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:12:29.724 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:12:29.724 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:12:29.724 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:12:29.724 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:12:29.725 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:12:29.725 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:12:29.725 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:12:29.725 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:12:29.725 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-26T00:12:29.725 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-26T00:12:29.725 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.0 2024-03-26T00:12:29.726 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:12:29.726 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-26T00:12:29.726 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-26T00:12:29.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:12:29.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:12:29.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:12:29.764 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:12:29.762+0000 7f7d13b2a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:12:29.766 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:12:29.766+0000 7f7d13b2a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:12:29.768 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:12:29.766+0000 7f7d13b2a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:12:30.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-26T00:12:30.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:12:30.049 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-26T00:12:30.049 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:12:30.049 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:12:30.049 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:12:30.049 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:12:30.049 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:12:30.049 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:12:30.049 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:12:30.302 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:12:30.298+0000 7f7d13b2a7c0 -1 Falling back to public interface 2024-03-26T00:12:30.380 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:12:31.381 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:12:31.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:12:31.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:12:31.382 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:12:31.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:12:31.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:12:31.700 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:12:32.702 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:12:32.702 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:12:32.702 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:12:32.703 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:12:32.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:12:32.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:12:33.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:12:33.022 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:12:33.022+0000 7f7d13b2a7c0 -1 osd.0 108 log_to_monitors true 2024-03-26T00:12:34.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:12:34.017 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:12:34.018 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:12:34.018 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:12:34.018 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:12:34.018 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:12:34.533 INFO:tasks.workunit.client.0.smithi129.stderr:osd.0 up in weight 1 up_from 112 up_thru 112 down_at 109 last_clean_interval [98,108) [v2:127.0.0.1:6802/953705469,v1:127.0.0.1:6803/953705469] [v2:127.0.0.1:6804/953705469,v1:127.0.0.1:6805/953705469] exists,up 27293645-99f8-4138-92f4-30c701220322 2024-03-26T00:12:34.534 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:12:34.534 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:12:34.534 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:12:34.534 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:12:34.534 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:12:34.534 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:12:34.534 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:12:34.535 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:12:34.535 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:12:34.535 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:12:34.535 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:12:34.535 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:12:34.535 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:12:34.640 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:12:34.641 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:12:34.641 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:12:34.641 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:12:34.641 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:12:34.641 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:12:34.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:12:34.991 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:12:34.991 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-26T00:12:34.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:12:34.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:12:34.992 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:12:35.137 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=481036337154 2024-03-26T00:12:35.137 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 481036337154 2024-03-26T00:12:35.137 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-481036337154' 2024-03-26T00:12:35.137 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:12:35.138 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:12:35.275 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=446676598789 2024-03-26T00:12:35.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 446676598789 2024-03-26T00:12:35.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-481036337154 1-446676598789' 2024-03-26T00:12:35.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:12:35.276 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:12:35.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383697 2024-03-26T00:12:35.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383697 2024-03-26T00:12:35.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-481036337154 1-446676598789 2-343597383697' 2024-03-26T00:12:35.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:12:35.419 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-481036337154 2024-03-26T00:12:35.419 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:12:35.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:12:35.421 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-481036337154 2024-03-26T00:12:35.421 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:12:35.422 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=481036337154 2024-03-26T00:12:35.422 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 481036337154' 2024-03-26T00:12:35.422 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 481036337154 2024-03-26T00:12:35.422 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:12:35.715 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 481036337154 2024-03-26T00:12:35.715 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:12:36.717 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:12:36.717 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:12:37.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 481036337154 -lt 481036337154 2024-03-26T00:12:37.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:12:37.037 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-446676598789 2024-03-26T00:12:37.038 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:12:37.039 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:12:37.040 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-446676598789 2024-03-26T00:12:37.040 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:12:37.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=446676598789 2024-03-26T00:12:37.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 446676598789' 2024-03-26T00:12:37.042 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 446676598789 2024-03-26T00:12:37.042 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:12:37.372 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 446676598790 -lt 446676598789 2024-03-26T00:12:37.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:12:37.374 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383697 2024-03-26T00:12:37.374 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:12:37.375 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:12:37.376 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383697 2024-03-26T00:12:37.376 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:12:37.378 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383697 2024-03-26T00:12:37.378 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383697' 2024-03-26T00:12:37.378 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 343597383697 2024-03-26T00:12:37.379 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:12:37.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383697 -lt 343597383697 2024-03-26T00:12:37.710 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:12:37.711 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:12:37.711 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:12:38.060 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:12:38.092 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:12:38.094 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:12:38.094 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:12:38.094 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:12:38.094 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:12:38.094 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:12:38.094 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:12:38.383 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:12:38.384 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:12:38.385 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:12:38.385 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:12:38.751 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:12:38.751 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:12:38.751 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:12:38.751 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3669: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-03-26T00:12:38.751 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:12:38.751 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:12:38.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-26T00:12:38.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:12:38.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-03-26T00:12:38.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:12:38.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:12:38.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-26T00:12:38.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:12:38.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-26T00:12:38.752 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:12:38.752 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:12:38.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:12:38.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:12:38.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:12:39.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:12:39.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-03-26T00:12:39.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:12:39.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:12:39.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:12:39.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:12:39.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:12:39.063 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-03-26T00:12:41.808 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-26T00:12:41.808 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:12:41.808 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:12:41.808 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-26T00:12:41.808 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:12:41.808 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:12:41.809 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:12:41.809 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:12:41.809 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:12:41.809 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:12:41.809 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-26T00:12:41.809 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-26T00:12:41.809 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:12:41.809 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:12:41.809 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:12:41.811 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:12:41.811 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:12:41.811 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:12:41.811 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:12:41.811 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:12:41.811 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:12:41.811 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:12:41.812 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:12:41.812 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:12:41.812 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:12:41.812 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:12:41.812 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:12:41.812 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:12:41.812 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:12:41.812 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:12:41.812 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:12:41.812 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:12:41.812 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-26T00:12:41.813 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-26T00:12:41.813 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.0 2024-03-26T00:12:41.813 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:12:41.813 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-26T00:12:41.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-26T00:12:41.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:12:41.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:12:41.815 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:12:41.846 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:12:41.845+0000 7f469a4707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:12:41.861 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:12:41.861+0000 7f469a4707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:12:41.862 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:12:41.865+0000 7f469a4707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:12:42.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-26T00:12:42.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:12:42.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-26T00:12:42.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:12:42.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:12:42.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:12:42.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:12:42.146 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:12:42.146 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:12:42.146 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:12:42.397 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:12:42.397+0000 7f469a4707c0 -1 Falling back to public interface 2024-03-26T00:12:42.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:12:43.482 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:12:43.482 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:12:43.482 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:12:43.482 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:12:43.483 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:12:43.483 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:12:43.799 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:12:44.801 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:12:44.801 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:12:44.801 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:12:44.802 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:12:44.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:12:44.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:12:45.033 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:12:45.033+0000 7f469a4707c0 -1 osd.0 114 log_to_monitors true 2024-03-26T00:12:45.167 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:12:46.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:12:46.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:12:46.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:12:46.169 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:12:46.170 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:12:46.170 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:12:46.598 INFO:tasks.workunit.client.0.smithi129.stderr:osd.0 up in weight 1 up_from 119 up_thru 119 down_at 115 last_clean_interval [112,114) [v2:127.0.0.1:6802/125327725,v1:127.0.0.1:6803/125327725] [v2:127.0.0.1:6804/125327725,v1:127.0.0.1:6805/125327725] exists,up 27293645-99f8-4138-92f4-30c701220322 2024-03-26T00:12:46.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:12:46.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:12:46.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:12:46.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:12:46.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:12:46.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:12:46.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:12:46.599 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:12:46.599 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:12:46.599 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:12:46.600 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:12:46.600 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:12:46.600 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:12:46.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:12:46.714 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:12:46.714 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:12:46.714 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:12:46.714 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:12:46.714 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:12:47.200 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:12:47.200 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:12:47.200 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-26T00:12:47.200 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:12:47.200 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:12:47.200 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:12:47.397 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=511101108226 2024-03-26T00:12:47.398 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 511101108226 2024-03-26T00:12:47.398 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-511101108226' 2024-03-26T00:12:47.398 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:12:47.398 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:12:47.540 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=446676598793 2024-03-26T00:12:47.540 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 446676598793 2024-03-26T00:12:47.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-511101108226 1-446676598793' 2024-03-26T00:12:47.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:12:47.541 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:12:47.683 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383700 2024-03-26T00:12:47.683 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383700 2024-03-26T00:12:47.684 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-511101108226 1-446676598793 2-343597383700' 2024-03-26T00:12:47.684 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:12:47.684 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-511101108226 2024-03-26T00:12:47.684 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:12:47.685 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:12:47.686 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-511101108226 2024-03-26T00:12:47.686 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:12:47.687 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=511101108226 2024-03-26T00:12:47.687 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 511101108226' 2024-03-26T00:12:47.687 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 511101108226 2024-03-26T00:12:47.687 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:12:48.049 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 511101108226 -lt 511101108226 2024-03-26T00:12:48.049 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:12:48.051 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-446676598793 2024-03-26T00:12:48.051 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:12:48.052 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:12:48.053 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-446676598793 2024-03-26T00:12:48.053 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:12:48.054 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=446676598793 2024-03-26T00:12:48.054 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 446676598793' 2024-03-26T00:12:48.054 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 446676598793 2024-03-26T00:12:48.054 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:12:48.399 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 446676598793 -lt 446676598793 2024-03-26T00:12:48.400 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:12:48.400 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383700 2024-03-26T00:12:48.401 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:12:48.402 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:12:48.403 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383700 2024-03-26T00:12:48.403 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:12:48.404 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383700 2024-03-26T00:12:48.404 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383700' 2024-03-26T00:12:48.404 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.2 seq 343597383700 2024-03-26T00:12:48.404 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:12:48.748 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383700 -lt 343597383700 2024-03-26T00:12:48.748 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:12:48.749 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:12:48.749 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:12:49.143 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:12:49.143 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:12:49.143 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:12:49.143 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:12:49.143 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:12:49.143 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:12:49.144 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:12:49.144 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:12:49.474 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:12:49.475 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:12:49.475 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:12:49.475 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:12:49.848 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:12:49.848 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:12:49.848 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:12:49.849 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3670: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2024-03-26T00:12:49.850 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2024-03-26T00:12:49.850 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-26T00:12:49.850 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=EOBJ0 2024-03-26T00:12:49.850 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-26T00:12:49.850 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool EOBJ0 2024-03-26T00:12:50.181 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: local pg=3.0 2024-03-26T00:12:50.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3678: corrupt_scrub_erasure: pg_scrub 3.0 2024-03-26T00:12:50.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-03-26T00:12:50.182 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-03-26T00:12:50.182 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:12:50.183 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:12:50.186 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:12:50.186 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-26T00:12:50.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-03-26T00:10:11.454998+0000 2024-03-26T00:12:50.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-03-26T00:12:50.787 INFO:tasks.workunit.client.0.smithi129.stderr:instructing pg 3.0s0 on osd.1 to scrub 2024-03-26T00:12:50.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-03-26T00:10:11.454998+0000 2024-03-26T00:12:50.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-26T00:12:50.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-26T00:10:11.454998+0000 2024-03-26T00:12:50.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-26T00:12:50.804 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-26T00:12:50.804 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:12:50.804 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-26T00:12:50.804 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:12:50.804 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:12:50.805 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:12:50.805 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-26T00:12:51.099 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:10:11.454998+0000 '>' 2024-03-26T00:10:11.454998+0000 2024-03-26T00:12:51.099 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:12:52.100 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:12:52.101 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:12:52.102 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-26T00:12:52.102 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:12:52.102 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:12:52.102 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:12:52.103 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-26T00:12:52.422 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:10:11.454998+0000 '>' 2024-03-26T00:10:11.454998+0000 2024-03-26T00:12:52.423 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:12:53.424 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:12:53.424 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:12:53.425 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-26T00:12:53.425 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:12:53.425 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:12:53.426 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:12:53.426 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-26T00:12:53.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:10:11.454998+0000 '>' 2024-03-26T00:10:11.454998+0000 2024-03-26T00:12:53.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:12:54.741 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:12:54.741 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:12:54.742 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-26T00:12:54.742 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:12:54.742 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:12:54.742 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:12:54.742 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-26T00:12:55.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:10:11.454998+0000 '>' 2024-03-26T00:10:11.454998+0000 2024-03-26T00:12:55.063 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:12:56.064 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:12:56.064 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:12:56.065 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-26T00:12:56.065 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:12:56.065 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:12:56.066 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:12:56.066 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-26T00:12:56.383 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:12:50.977720+0000 '>' 2024-03-26T00:10:11.454998+0000 2024-03-26T00:12:56.384 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-26T00:12:56.384 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3680: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-03-26T00:12:56.446 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-03-26T00:12:56.476 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: test 1 = 1 2024-03-26T00:12:56.476 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-03-26T00:12:56.506 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: test 3.0 = 3.0 2024-03-26T00:12:56.506 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3686: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-03-26T00:12:56.534 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-03-26T00:12:56.563 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: epoch=119 2024-03-26T00:12:56.563 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq 'def walk(f): 2024-03-26T00:12:56.563 INFO:tasks.workunit.client.0.smithi129.stderr: . as $in 2024-03-26T00:12:56.563 INFO:tasks.workunit.client.0.smithi129.stderr: | if type == "object" then 2024-03-26T00:12:56.564 INFO:tasks.workunit.client.0.smithi129.stderr: reduce keys[] as $key 2024-03-26T00:12:56.564 INFO:tasks.workunit.client.0.smithi129.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-26T00:12:56.564 INFO:tasks.workunit.client.0.smithi129.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-26T00:12:56.564 INFO:tasks.workunit.client.0.smithi129.stderr: else f 2024-03-26T00:12:56.564 INFO:tasks.workunit.client.0.smithi129.stderr: end; 2024-03-26T00:12:56.564 INFO:tasks.workunit.client.0.smithi129.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-26T00:12:56.564 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-26T00:12:56.564 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-26T00:12:56.564 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-26T00:12:56.564 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-03-26T00:12:56.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq .inconsistents 2024-03-26T00:12:56.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-26T00:12:56.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq 'def walk(f): 2024-03-26T00:12:56.608 INFO:tasks.workunit.client.0.smithi129.stderr: . as $in 2024-03-26T00:12:56.608 INFO:tasks.workunit.client.0.smithi129.stderr: | if type == "object" then 2024-03-26T00:12:56.608 INFO:tasks.workunit.client.0.smithi129.stderr: reduce keys[] as $key 2024-03-26T00:12:56.608 INFO:tasks.workunit.client.0.smithi129.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-26T00:12:56.608 INFO:tasks.workunit.client.0.smithi129.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-26T00:12:56.608 INFO:tasks.workunit.client.0.smithi129.stderr: else f 2024-03-26T00:12:56.608 INFO:tasks.workunit.client.0.smithi129.stderr: end; 2024-03-26T00:12:56.608 INFO:tasks.workunit.client.0.smithi129.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-26T00:12:56.608 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-26T00:12:56.608 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-26T00:12:56.608 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-26T00:12:56.608 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-03-26T00:12:56.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq .inconsistents 2024-03-26T00:12:56.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-26T00:12:56.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4311: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-26T00:12:56.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-26T00:12:56.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4312: corrupt_scrub_erasure: test no = yes 2024-03-26T00:12:56.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4317: corrupt_scrub_erasure: test '' = yes 2024-03-26T00:12:56.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4322: corrupt_scrub_erasure: pg_deep_scrub 3.0 2024-03-26T00:12:56.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-03-26T00:12:56.651 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-26T00:12:56.651 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:12:56.651 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-26T00:12:56.652 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:12:56.652 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-26T00:12:56.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-03-26T00:10:11.454998+0000 2024-03-26T00:12:56.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-03-26T00:12:57.256 INFO:tasks.workunit.client.0.smithi129.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2024-03-26T00:12:57.273 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-03-26T00:10:11.454998+0000 last_deep_scrub_stamp 2024-03-26T00:12:57.273 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-26T00:12:57.273 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-26T00:10:11.454998+0000 2024-03-26T00:12:57.273 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-03-26T00:12:57.273 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-26T00:12:57.273 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:12:57.274 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-26T00:12:57.274 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:12:57.274 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-26T00:12:57.274 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:12:57.275 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-26T00:12:57.643 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:10:11.454998+0000 '>' 2024-03-26T00:10:11.454998+0000 2024-03-26T00:12:57.644 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:12:58.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:12:58.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:12:58.646 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-26T00:12:58.647 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:12:58.647 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-26T00:12:58.647 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:12:58.647 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-26T00:12:58.954 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:10:11.454998+0000 '>' 2024-03-26T00:10:11.454998+0000 2024-03-26T00:12:58.954 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:12:59.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:12:59.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:12:59.957 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-26T00:12:59.957 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:12:59.957 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-26T00:12:59.957 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:12:59.957 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-26T00:13:00.275 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:10:11.454998+0000 '>' 2024-03-26T00:10:11.454998+0000 2024-03-26T00:13:00.275 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:13:01.277 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:13:01.277 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:13:01.278 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-26T00:13:01.278 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:13:01.278 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-26T00:13:01.279 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:13:01.279 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-26T00:13:01.597 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:10:11.454998+0000 '>' 2024-03-26T00:10:11.454998+0000 2024-03-26T00:13:01.597 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:13:02.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:13:02.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:13:02.599 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-26T00:13:02.599 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:13:02.600 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-26T00:13:02.600 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:13:02.600 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-26T00:13:02.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:12:58.267890+0000 '>' 2024-03-26T00:10:11.454998+0000 2024-03-26T00:13:02.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-26T00:13:02.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4324: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-03-26T00:13:02.947 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-03-26T00:13:02.975 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: test 1 = 1 2024-03-26T00:13:02.975 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-03-26T00:13:03.004 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: test 3.0 = 3.0 2024-03-26T00:13:03.004 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4330: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-03-26T00:13:03.031 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-03-26T00:13:03.059 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: epoch=119 2024-03-26T00:13:03.059 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4334: corrupt_scrub_erasure: '[' true = true ']' 2024-03-26T00:13:03.059 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: jq 'def walk(f): 2024-03-26T00:13:03.059 INFO:tasks.workunit.client.0.smithi129.stderr: . as $in 2024-03-26T00:13:03.059 INFO:tasks.workunit.client.0.smithi129.stderr: | if type == "object" then 2024-03-26T00:13:03.059 INFO:tasks.workunit.client.0.smithi129.stderr: reduce keys[] as $key 2024-03-26T00:13:03.059 INFO:tasks.workunit.client.0.smithi129.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-26T00:13:03.059 INFO:tasks.workunit.client.0.smithi129.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-26T00:13:03.059 INFO:tasks.workunit.client.0.smithi129.stderr: else f 2024-03-26T00:13:03.060 INFO:tasks.workunit.client.0.smithi129.stderr: end; 2024-03-26T00:13:03.060 INFO:tasks.workunit.client.0.smithi129.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-26T00:13:03.060 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-26T00:13:03.060 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-26T00:13:03.060 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-26T00:13:03.060 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-03-26T00:13:03.060 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: jq .inconsistents 2024-03-26T00:13:03.060 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-26T00:13:03.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq 'def walk(f): 2024-03-26T00:13:03.102 INFO:tasks.workunit.client.0.smithi129.stderr: . as $in 2024-03-26T00:13:03.102 INFO:tasks.workunit.client.0.smithi129.stderr: | if type == "object" then 2024-03-26T00:13:03.102 INFO:tasks.workunit.client.0.smithi129.stderr: reduce keys[] as $key 2024-03-26T00:13:03.102 INFO:tasks.workunit.client.0.smithi129.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-26T00:13:03.103 INFO:tasks.workunit.client.0.smithi129.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-26T00:13:03.103 INFO:tasks.workunit.client.0.smithi129.stderr: else f 2024-03-26T00:13:03.103 INFO:tasks.workunit.client.0.smithi129.stderr: end; 2024-03-26T00:13:03.103 INFO:tasks.workunit.client.0.smithi129.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-26T00:13:03.103 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-26T00:13:03.103 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-26T00:13:03.103 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-26T00:13:03.103 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-03-26T00:13:03.103 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq .inconsistents 2024-03-26T00:13:03.103 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-26T00:13:03.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5725: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-26T00:13:03.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-26T00:13:03.143 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5726: corrupt_scrub_erasure: test no = yes 2024-03-26T00:13:03.144 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5737: corrupt_scrub_erasure: test '' = yes 2024-03-26T00:13:03.144 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5742: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2024-03-26T00:13:03.640 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'ecpool' removed 2024-03-26T00:13:03.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-26T00:13:03.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-26T00:13:03.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-26T00:13:03.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-26T00:13:03.662 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:13:03.662 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:13:03.663 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:13:03.663 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:13:03.663 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:13:03.975 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:13:03.976 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-26T00:13:03.978 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-26T00:13:03.979 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-26T00:13:03.980 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-26T00:13:03.980 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-26T00:13:03.981 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-26T00:13:03.982 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:13:03.982 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-26T00:13:03.982 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-26T00:13:03.983 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:13:03.984 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-26T00:13:03.986 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-26T00:13:03.986 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-26T00:13:04.021 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-26T00:13:04.021 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:13:04.021 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:13:04.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-26T00:13:04.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-26T00:13:04.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-26T00:13:04.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-26T00:13:04.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-26T00:13:04.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-26T00:13:04.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-26T00:13:04.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-26T00:13:04.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-26T00:13:04.024 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-26T00:13:04.024 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:13:04.024 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:13:04.024 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:13:04.024 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:13:04.024 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:13:04.026 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:13:04.027 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-26T00:13:04.028 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-26T00:13:04.028 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-26T00:13:04.029 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-26T00:13:04.029 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-26T00:13:04.030 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-26T00:13:04.031 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:13:04.031 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-26T00:13:04.031 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-26T00:13:04.032 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:13:04.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-26T00:13:04.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-26T00:13:04.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-26T00:13:04.036 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-26T00:13:04.036 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:13:04.036 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:13:04.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-26T00:13:04.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-26T00:13:04.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-26T00:13:04.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-26T00:13:04.038 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-26T00:13:04.039 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:13:04.039 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:13:04.039 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61636 2024-03-26T00:13:04.040 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-26T00:13:04.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-26T00:13:04.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-26T00:13:04.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-26T00:13:04.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_replicated td/osd-scrub-repair 2024-03-26T00:13:04.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:991: TEST_corrupt_scrub_replicated: local dir=td/osd-scrub-repair 2024-03-26T00:13:04.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:992: TEST_corrupt_scrub_replicated: local poolname=csr_pool 2024-03-26T00:13:04.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:993: TEST_corrupt_scrub_replicated: local total_objs=19 2024-03-26T00:13:04.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:995: TEST_corrupt_scrub_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-03-26T00:13:04.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-26T00:13:04.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-26T00:13:04.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-26T00:13:04.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-26T00:13:04.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-26T00:13:04.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-03-26T00:13:04.500 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-26T00:13:04.500 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:13:04.500 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:13:04.501 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:13:04.501 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:13:04.501 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:13:04.501 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:13:04.502 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-03-26T00:13:04.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-26T00:13:04.754 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-26T00:13:04.754 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-26T00:13:04.754 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-26T00:13:04.754 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-26T00:13:04.755 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-26T00:13:04.755 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-26T00:13:04.756 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-26T00:13:04.756 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-26T00:13:04.756 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-26T00:13:04.756 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:13:04.756 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:13:04.756 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-26T00:13:04.757 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-26T00:13:04.757 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get fsid 2024-03-26T00:13:04.871 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-26T00:13:04.871 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-26T00:13:04.871 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-26T00:13:04.871 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-26T00:13:04.872 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-26T00:13:04.872 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-26T00:13:04.872 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-26T00:13:04.872 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-26T00:13:04.873 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-26T00:13:04.873 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:13:04.873 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:13:04.874 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-26T00:13:04.874 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-26T00:13:04.874 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get mon_host 2024-03-26T00:13:04.975 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:996: TEST_corrupt_scrub_replicated: run_mgr td/osd-scrub-repair x 2024-03-26T00:13:04.975 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-26T00:13:04.975 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-26T00:13:04.975 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-26T00:13:04.975 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-26T00:13:04.975 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-26T00:13:04.975 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-26T00:13:05.885 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-26T00:13:05.885 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:13:05.885 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:13:05.886 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:13:05.886 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:13:05.886 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:13:05.886 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:13:05.887 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-26T00:13:05.888 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-26T00:13:05.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:997: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 0 2024-03-26T00:13:05.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-26T00:13:05.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-26T00:13:05.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-26T00:13:05.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-26T00:13:05.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:13:05.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:13:05.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:13:05.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:13:05.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:13:05.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-26T00:13:05.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-26T00:13:05.924 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-26T00:13:05.924 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-26T00:13:05.924 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:13:05.926 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-26T00:13:05.926 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:13:05.926 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:13:05.926 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:13:05.926 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:13:05.926 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:13:05.926 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:13:05.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:13:05.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:13:05.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-26T00:13:05.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-26T00:13:05.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:13:05.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:13:05.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:13:05.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:13:05.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:13:05.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:13:05.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-26T00:13:05.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-26T00:13:05.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-26T00:13:05.927 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-26T00:13:05.928 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d7a8ca59-f9ab-4120-b1c3-f87ea4b0bd36 2024-03-26T00:13:05.928 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d7a8ca59-f9ab-4120-b1c3-f87ea4b0bd36' 2024-03-26T00:13:05.928 INFO:tasks.workunit.client.0.smithi129.stdout:add osd0 d7a8ca59-f9ab-4120-b1c3-f87ea4b0bd36 2024-03-26T00:13:05.928 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-26T00:13:05.945 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAREwJm58x1OBAA+3CMIcpE4ulS390NuSyMwg== 2024-03-26T00:13:05.945 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAREwJm58x1OBAA+3CMIcpE4ulS390NuSyMwg=="}' 2024-03-26T00:13:05.945 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d7a8ca59-f9ab-4120-b1c3-f87ea4b0bd36 -i td/osd-scrub-repair/0/new.json 2024-03-26T00:13:06.265 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-26T00:13:06.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-26T00:13:06.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAREwJm58x1OBAA+3CMIcpE4ulS390NuSyMwg== --osd-uuid d7a8ca59-f9ab-4120-b1c3-f87ea4b0bd36 2024-03-26T00:13:06.319 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:13:06.321+0000 7f1cb24c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:13:06.358 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:13:06.357+0000 7f1cb24c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:13:06.359 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:13:06.361+0000 7f1cb24c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:13:06.360 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:13:06.361+0000 7f1cb24c97c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-26T00:13:06.360 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:13:06.361+0000 7f1cb24c97c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-26T00:13:06.360 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:13:06.361+0000 7f1cb24c97c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-26T00:13:06.491 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:13:06.493+0000 7f1cb24c97c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-26T00:13:12.852 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-26T00:13:12.852 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-26T00:13:12.855 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-26T00:13:12.855 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd0 key to auth repository 2024-03-26T00:13:12.856 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-26T00:13:13.248 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.0 2024-03-26T00:13:13.249 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-26T00:13:13.250 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:13:13.250 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-26T00:13:13.250 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-26T00:13:13.250 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-26T00:13:13.281 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:13:13.281+0000 7fdab920b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:13:13.324 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:13:13.325+0000 7fdab920b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:13:13.326 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:13:13.325+0000 7fdab920b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:13:13.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-26T00:13:13.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:13:13.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-26T00:13:13.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:13:13.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:13:13.561 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-26T00:13:13.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:13:13.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:13:13.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:13:13.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:13:13.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:13:13.928 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:13:13.929+0000 7fdab920b7c0 -1 Falling back to public interface 2024-03-26T00:13:14.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:13:14.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:13:14.873 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-26T00:13:14.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:13:14.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:13:14.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:13:15.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:13:16.180 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:13:16.181 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:13:16.181 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-26T00:13:16.181 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:13:16.181 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:13:16.181 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:13:16.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:13:16.508 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:13:16.509+0000 7fdab920b7c0 -1 osd.0 0 log_to_monitors true 2024-03-26T00:13:17.501 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:13:17.501 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:13:17.501 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-26T00:13:17.502 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:13:17.502 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:13:17.502 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:13:17.852 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:13:18.854 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:13:18.854 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:13:18.854 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-26T00:13:18.855 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-26T00:13:18.855 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:13:18.855 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:13:19.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:13:20.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:13:20.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:13:20.177 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-26T00:13:20.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-26T00:13:20.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:13:20.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:13:20.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:13:21.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:13:21.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:13:21.651 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-26T00:13:21.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-26T00:13:21.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:13:21.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:13:21.971 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:13:22.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:13:22.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:13:22.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-26T00:13:22.974 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-26T00:13:22.974 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:13:22.974 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:13:23.295 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:13:24.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:13:24.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:13:24.297 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-26T00:13:24.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-26T00:13:24.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:13:24.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:13:24.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:13:25.628 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:13:25.629 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:13:25.629 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-26T00:13:25.629 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-26T00:13:25.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:13:25.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:13:25.946 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:13:26.947 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:13:26.947 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:13:26.948 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-26T00:13:26.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-26T00:13:26.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:13:26.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:13:27.270 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:13:28.272 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:13:28.272 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:13:28.272 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-26T00:13:28.273 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-26T00:13:28.273 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:13:28.273 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:13:28.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:13:29.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:13:29.662 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:13:29.662 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-26T00:13:29.662 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-26T00:13:29.663 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:13:29.663 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:13:29.985 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:13:30.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:13:30.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:13:30.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-26T00:13:30.988 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-26T00:13:30.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:13:30.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:13:31.606 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:13:32.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:13:32.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:13:32.608 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-26T00:13:32.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-26T00:13:32.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:13:32.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:13:33.548 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:13:34.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:13:34.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:13:34.551 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-26T00:13:34.551 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-26T00:13:34.551 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:13:34.551 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:13:35.519 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:13:36.520 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:13:36.521 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:13:36.521 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-26T00:13:36.521 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-26T00:13:36.521 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:13:36.522 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:13:36.958 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:13:37.960 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:13:37.960 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:13:37.961 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-26T00:13:37.961 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-26T00:13:37.961 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:13:37.961 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:13:38.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:13:39.672 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:13:39.672 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:13:39.672 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-26T00:13:39.673 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-26T00:13:39.673 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:13:39.673 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:13:40.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:13:41.414 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:13:41.414 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:13:41.414 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-26T00:13:41.415 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-26T00:13:41.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:13:41.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:13:41.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:13:42.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:13:42.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:13:42.784 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-26T00:13:42.785 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-26T00:13:42.785 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:13:42.785 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:13:43.228 INFO:tasks.workunit.client.0.smithi129.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2241128351,v1:127.0.0.1:6803/2241128351] [v2:127.0.0.1:6804/2241128351,v1:127.0.0.1:6805/2241128351] exists,up d7a8ca59-f9ab-4120-b1c3-f87ea4b0bd36 2024-03-26T00:13:43.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:13:43.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:13:43.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:13:43.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:998: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 1 2024-03-26T00:13:43.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-26T00:13:43.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-26T00:13:43.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-26T00:13:43.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-26T00:13:43.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:13:43.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:13:43.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:13:43.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:13:43.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:13:43.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:13:43.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:13:43.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-26T00:13:43.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-26T00:13:43.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:13:43.232 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-26T00:13:43.232 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:13:43.232 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:13:43.234 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:13:43.234 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:13:43.234 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:13:43.234 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:13:43.234 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:13:43.234 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:13:43.234 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-26T00:13:43.234 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-26T00:13:43.234 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:13:43.234 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:13:43.234 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:13:43.234 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:13:43.235 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:13:43.235 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:13:43.235 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-26T00:13:43.235 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-26T00:13:43.235 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:13:43.235 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-26T00:13:43.235 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0c2624cc-3bc6-4489-918a-a2155ff79e95 2024-03-26T00:13:43.236 INFO:tasks.workunit.client.0.smithi129.stdout:add osd1 0c2624cc-3bc6-4489-918a-a2155ff79e95 2024-03-26T00:13:43.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 0c2624cc-3bc6-4489-918a-a2155ff79e95' 2024-03-26T00:13:43.236 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-26T00:13:43.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA3EwJmVbl0DxAA58Kmr+IINPuMrVoIuNjWcw== 2024-03-26T00:13:43.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA3EwJmVbl0DxAA58Kmr+IINPuMrVoIuNjWcw=="}' 2024-03-26T00:13:43.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0c2624cc-3bc6-4489-918a-a2155ff79e95 -i td/osd-scrub-repair/1/new.json 2024-03-26T00:13:43.775 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-26T00:13:43.795 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-26T00:13:43.796 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA3EwJmVbl0DxAA58Kmr+IINPuMrVoIuNjWcw== --osd-uuid 0c2624cc-3bc6-4489-918a-a2155ff79e95 2024-03-26T00:13:43.831 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:13:43.832+0000 7f1d0743d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:13:43.911 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:13:43.912+0000 7f1d0743d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:13:43.913 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:13:43.912+0000 7f1d0743d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:13:43.913 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:13:43.912+0000 7f1d0743d7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-26T00:13:43.913 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:13:43.912+0000 7f1d0743d7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-26T00:13:43.914 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:13:43.912+0000 7f1d0743d7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-26T00:13:44.004 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:13:44.004+0000 7f1d0743d7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-26T00:13:51.233 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-26T00:13:51.233 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-26T00:13:51.235 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd1 key to auth repository 2024-03-26T00:13:51.235 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-26T00:13:51.235 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-26T00:13:51.628 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.1 2024-03-26T00:13:51.629 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-26T00:13:51.629 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:13:51.629 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-26T00:13:51.629 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-26T00:13:51.629 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-26T00:13:51.655 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:13:51.656+0000 7fe06debc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:13:51.657 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:13:51.656+0000 7fe06debc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:13:51.658 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:13:51.660+0000 7fe06debc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:13:51.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-26T00:13:51.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:13:51.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:13:51.930 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-26T00:13:51.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:13:51.931 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:13:51.931 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:13:51.931 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:13:51.931 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:13:51.931 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:13:52.192 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:13:52.192+0000 7fe06debc7c0 -1 Falling back to public interface 2024-03-26T00:13:52.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:13:53.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:13:53.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:13:53.232 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-26T00:13:53.233 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:13:53.233 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:13:53.233 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:13:53.538 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:13:54.540 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:13:54.540 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:13:54.540 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-26T00:13:54.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:13:54.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:13:54.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:13:54.813 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:13:54.812+0000 7fe06debc7c0 -1 osd.1 0 log_to_monitors true 2024-03-26T00:13:54.862 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:13:55.864 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:13:55.864 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:13:55.864 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-26T00:13:55.865 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:13:55.865 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:13:55.865 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:13:56.338 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:13:57.339 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:13:57.340 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:13:57.340 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-26T00:13:57.340 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-26T00:13:57.340 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:13:57.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:13:57.715 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:13:58.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:13:58.717 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:13:58.717 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-26T00:13:58.717 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-26T00:13:58.717 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:13:58.717 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:13:59.077 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:14:00.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:14:00.079 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:14:00.079 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-26T00:14:00.080 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-26T00:14:00.080 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:14:00.080 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:14:00.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:14:01.396 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:14:01.396 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:14:01.396 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-26T00:14:01.397 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-26T00:14:01.397 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:14:01.397 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:14:01.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:14:02.989 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:14:02.989 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:14:02.989 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-26T00:14:02.990 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-26T00:14:02.990 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:14:02.990 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:14:03.380 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:14:04.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:14:04.383 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:14:04.383 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-26T00:14:04.383 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-26T00:14:04.383 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:14:04.383 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:14:04.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:14:05.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:14:05.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:14:05.802 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-26T00:14:05.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-26T00:14:05.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:14:05.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:14:06.127 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:14:07.128 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:14:07.129 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:14:07.129 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-26T00:14:07.129 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-26T00:14:07.130 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:14:07.130 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:14:07.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:14:08.647 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:14:08.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:14:08.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-26T00:14:08.648 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-26T00:14:08.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:14:08.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:14:09.031 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:14:10.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:14:10.035 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:14:10.035 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-26T00:14:10.035 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-26T00:14:10.035 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:14:10.035 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:14:10.362 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:14:11.363 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:14:11.363 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:14:11.364 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-26T00:14:11.364 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-26T00:14:11.364 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:14:11.364 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:14:11.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:14:12.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:14:12.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:14:12.833 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-26T00:14:12.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-26T00:14:12.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:14:12.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:14:13.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:14:14.652 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:14:14.652 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:14:14.652 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-26T00:14:14.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-26T00:14:14.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:14:14.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:14:15.101 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:14:16.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:14:16.103 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:14:16.103 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-26T00:14:16.103 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-26T00:14:16.104 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:14:16.104 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:14:16.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:14:17.705 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:14:17.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:14:17.706 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-26T00:14:17.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-26T00:14:17.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:14:17.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:14:18.375 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:14:19.376 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:14:19.377 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:14:19.377 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-26T00:14:19.377 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-26T00:14:19.377 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:14:19.377 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:14:20.035 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:14:21.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:14:21.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:14:21.037 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-26T00:14:21.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-26T00:14:21.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:14:21.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:14:21.526 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:14:22.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:14:22.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:14:22.528 INFO:tasks.workunit.client.0.smithi129.stdout:21 2024-03-26T00:14:22.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-26T00:14:22.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:14:22.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:14:22.864 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:14:23.866 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:14:23.866 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:14:23.867 INFO:tasks.workunit.client.0.smithi129.stdout:22 2024-03-26T00:14:23.867 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-26T00:14:23.867 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:14:23.867 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:14:24.193 INFO:tasks.workunit.client.0.smithi129.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3499578400,v1:127.0.0.1:6811/3499578400] [v2:127.0.0.1:6812/3499578400,v1:127.0.0.1:6813/3499578400] exists,up 0c2624cc-3bc6-4489-918a-a2155ff79e95 2024-03-26T00:14:24.193 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:14:24.193 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:14:24.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:14:24.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:999: TEST_corrupt_scrub_replicated: create_rbd_pool 2024-03-26T00:14:24.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-26T00:14:24.554 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'rbd' does not exist 2024-03-26T00:14:24.570 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-26T00:14:24.570 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-26T00:14:25.206 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'rbd' created 2024-03-26T00:14:25.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-26T00:14:26.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-26T00:14:28.542 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1000: TEST_corrupt_scrub_replicated: wait_for_clean 2024-03-26T00:14:28.542 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:14:28.542 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:14:28.543 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:14:28.543 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:14:28.545 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:14:28.545 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:14:28.545 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:14:28.545 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:14:28.545 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:14:28.673 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:14:28.673 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:14:28.673 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:14:28.673 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:14:28.673 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:14:28.674 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:14:28.983 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:14:28.983 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:14:28.983 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:14:28.983 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:14:28.984 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:14:29.147 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836495 2024-03-26T00:14:29.148 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836495 2024-03-26T00:14:29.148 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836495' 2024-03-26T00:14:29.148 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:14:29.148 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:14:29.295 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672965 2024-03-26T00:14:29.295 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672965 2024-03-26T00:14:29.295 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836495 1-42949672965' 2024-03-26T00:14:29.295 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:14:29.295 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836495 2024-03-26T00:14:29.295 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:14:29.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:14:29.297 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836495 2024-03-26T00:14:29.298 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:14:29.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836495 2024-03-26T00:14:29.299 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 21474836495 2024-03-26T00:14:29.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2024-03-26T00:14:29.299 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:14:29.600 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836494 -lt 21474836495 2024-03-26T00:14:29.600 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:14:30.602 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:14:30.603 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:14:30.913 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836495 -lt 21474836495 2024-03-26T00:14:30.913 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:14:30.913 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:14:30.914 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672965 2024-03-26T00:14:30.915 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:14:30.916 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672965 2024-03-26T00:14:30.916 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:14:30.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672965 2024-03-26T00:14:30.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-03-26T00:14:30.917 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 42949672965 2024-03-26T00:14:30.917 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:14:31.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672965 -lt 42949672965 2024-03-26T00:14:31.332 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:14:31.333 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:14:31.333 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:14:31.775 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-26T00:14:31.775 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:14:31.776 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:14:31.776 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:14:31.776 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:14:31.776 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:14:31.776 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:14:31.776 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:14:32.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-26T00:14:32.069 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:14:32.070 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:14:32.070 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:14:32.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-26T00:14:32.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:14:32.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:14:32.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1002: TEST_corrupt_scrub_replicated: create_pool foo 1 2024-03-26T00:14:32.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-03-26T00:14:33.036 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'foo' created 2024-03-26T00:14:33.060 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-26T00:14:34.061 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1003: TEST_corrupt_scrub_replicated: create_pool csr_pool 1 1 2024-03-26T00:14:34.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create csr_pool 1 1 2024-03-26T00:14:34.668 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'csr_pool' created 2024-03-26T00:14:34.686 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-26T00:14:35.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1004: TEST_corrupt_scrub_replicated: wait_for_clean 2024-03-26T00:14:35.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:14:35.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:14:35.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:14:35.690 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:14:35.691 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:14:35.691 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:14:35.691 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:14:35.691 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:14:35.691 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:14:35.825 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:14:35.826 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:14:35.826 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:14:35.826 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:14:35.826 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:14:35.826 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:14:36.204 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:14:36.204 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:14:36.204 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:14:36.204 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:14:36.205 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:14:36.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836497 2024-03-26T00:14:36.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836497 2024-03-26T00:14:36.353 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497' 2024-03-26T00:14:36.353 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:14:36.353 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:14:36.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672967 2024-03-26T00:14:36.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672967 2024-03-26T00:14:36.495 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497 1-42949672967' 2024-03-26T00:14:36.495 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:14:36.495 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:14:36.495 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836497 2024-03-26T00:14:36.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:14:36.497 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836497 2024-03-26T00:14:36.497 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:14:36.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836497 2024-03-26T00:14:36.499 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 21474836497 2024-03-26T00:14:36.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2024-03-26T00:14:36.499 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:14:36.824 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836497 2024-03-26T00:14:36.825 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:14:37.826 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:14:37.827 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:14:38.150 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836497 2024-03-26T00:14:38.150 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:14:38.151 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672967 2024-03-26T00:14:38.151 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:14:38.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:14:38.154 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672967 2024-03-26T00:14:38.154 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:14:38.155 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 42949672967 2024-03-26T00:14:38.156 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672967 2024-03-26T00:14:38.156 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-03-26T00:14:38.156 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:14:38.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672966 -lt 42949672967 2024-03-26T00:14:38.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:14:39.506 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-26T00:14:39.507 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:14:39.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672967 -lt 42949672967 2024-03-26T00:14:39.815 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:14:39.815 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:14:39.816 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:14:40.174 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:14:40.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:14:40.175 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:14:40.175 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:14:40.176 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:14:40.176 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:14:40.176 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:14:40.176 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:14:40.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:14:40.529 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:14:40.529 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:14:40.529 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:14:40.883 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:14:40.883 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:14:40.883 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:14:40.883 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: seq 1 19 2024-03-26T00:14:40.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-26T00:14:40.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ1 2024-03-26T00:14:40.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ1 2024-03-26T00:14:40.885 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-26T00:14:40.885 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-26T00:14:40.885 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ1 2024-03-26T00:14:40.885 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-26T00:14:40.885 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-26T00:14:40.885 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-26T00:14:41.356 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-26T00:14:41.376 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-26T00:14:41.857 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-26T00:14:41.879 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-26T00:14:41.879 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-26T00:14:41.879 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:14:42.214 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2024-03-26T00:14:42.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2024-03-26T00:14:42.657 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-26T00:14:42.658 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ2 2024-03-26T00:14:42.658 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ2 2024-03-26T00:14:42.658 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-26T00:14:42.658 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-26T00:14:42.658 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ2 2024-03-26T00:14:42.658 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-26T00:14:42.658 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-26T00:14:42.658 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-26T00:14:43.204 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-26T00:14:43.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-26T00:14:43.697 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-26T00:14:43.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-26T00:14:43.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-26T00:14:43.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:14:44.044 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2024-03-26T00:14:44.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2024-03-26T00:14:44.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-26T00:14:44.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ3 2024-03-26T00:14:44.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ3 2024-03-26T00:14:44.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-26T00:14:44.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-26T00:14:44.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ3 2024-03-26T00:14:44.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-26T00:14:44.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-26T00:14:44.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-26T00:14:44.909 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-26T00:14:44.934 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-26T00:14:45.403 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-26T00:14:45.424 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-26T00:14:45.424 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-26T00:14:45.424 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ3 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:14:45.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ3 hdr-ROBJ3 2024-03-26T00:14:45.977 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ3 key-ROBJ3 val-ROBJ3 2024-03-26T00:14:46.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-26T00:14:46.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ4 2024-03-26T00:14:46.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ4 2024-03-26T00:14:46.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-26T00:14:46.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-26T00:14:46.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ4 2024-03-26T00:14:46.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-26T00:14:46.087 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-26T00:14:46.087 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-26T00:14:46.530 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-26T00:14:46.551 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-26T00:14:47.066 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-26T00:14:47.087 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-26T00:14:47.087 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-26T00:14:47.087 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ4 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:14:47.473 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ4 hdr-ROBJ4 2024-03-26T00:14:47.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ4 key-ROBJ4 val-ROBJ4 2024-03-26T00:14:47.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-26T00:14:47.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ5 2024-03-26T00:14:47.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ5 2024-03-26T00:14:47.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-26T00:14:47.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-26T00:14:47.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ5 2024-03-26T00:14:47.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-26T00:14:47.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-26T00:14:47.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-26T00:14:48.203 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-26T00:14:48.222 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-26T00:14:48.839 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-26T00:14:48.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-26T00:14:48.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-26T00:14:48.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ5 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:14:49.152 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ5 hdr-ROBJ5 2024-03-26T00:14:49.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ5 key-ROBJ5 val-ROBJ5 2024-03-26T00:14:49.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-26T00:14:49.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ6 2024-03-26T00:14:49.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ6 2024-03-26T00:14:49.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-26T00:14:49.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-26T00:14:49.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ6 2024-03-26T00:14:49.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-26T00:14:49.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-26T00:14:49.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-26T00:14:50.026 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-26T00:14:50.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-26T00:14:50.519 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-26T00:14:50.542 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-26T00:14:50.542 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-26T00:14:50.542 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ6 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:14:50.859 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ6 hdr-ROBJ6 2024-03-26T00:14:51.068 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ6 key-ROBJ6 val-ROBJ6 2024-03-26T00:14:51.202 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-26T00:14:51.202 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ7 2024-03-26T00:14:51.202 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ7 2024-03-26T00:14:51.202 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-26T00:14:51.202 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-26T00:14:51.202 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ7 2024-03-26T00:14:51.202 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-26T00:14:51.202 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-26T00:14:51.202 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-26T00:14:51.648 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-26T00:14:51.668 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-26T00:14:52.141 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-26T00:14:52.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-26T00:14:52.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-26T00:14:52.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ7 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:14:52.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ7 hdr-ROBJ7 2024-03-26T00:14:52.714 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ7 key-ROBJ7 val-ROBJ7 2024-03-26T00:14:52.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-26T00:14:52.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ8 2024-03-26T00:14:52.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ8 2024-03-26T00:14:52.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-26T00:14:52.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-26T00:14:52.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ8 2024-03-26T00:14:52.918 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-26T00:14:52.918 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-26T00:14:52.918 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-26T00:14:53.363 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-26T00:14:53.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-26T00:14:53.873 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-26T00:14:53.893 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-26T00:14:53.894 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-26T00:14:53.894 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ8 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:14:54.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ8 hdr-ROBJ8 2024-03-26T00:14:54.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ8 key-ROBJ8 val-ROBJ8 2024-03-26T00:14:54.522 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-26T00:14:54.522 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-03-26T00:14:54.522 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ9 2024-03-26T00:14:54.523 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-26T00:14:54.523 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-26T00:14:54.523 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ9 2024-03-26T00:14:54.523 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-26T00:14:54.523 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-26T00:14:54.523 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-26T00:14:54.977 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-26T00:14:54.997 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-26T00:14:55.470 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-26T00:14:55.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-26T00:14:55.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-26T00:14:55.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:14:55.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ9 hdr-ROBJ9 2024-03-26T00:14:56.026 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ9 key-ROBJ9 val-ROBJ9 2024-03-26T00:14:56.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-26T00:14:56.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-03-26T00:14:56.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ10 2024-03-26T00:14:56.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-26T00:14:56.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-26T00:14:56.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ10 2024-03-26T00:14:56.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-26T00:14:56.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-26T00:14:56.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-26T00:14:56.573 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-26T00:14:56.593 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-26T00:14:57.092 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-26T00:14:57.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-26T00:14:57.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-26T00:14:57.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ10 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:14:57.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ10 hdr-ROBJ10 2024-03-26T00:14:57.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key-ROBJ10 val-ROBJ10 2024-03-26T00:14:57.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-26T00:14:57.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ11 2024-03-26T00:14:57.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ11 2024-03-26T00:14:57.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-26T00:14:57.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-26T00:14:57.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ11 2024-03-26T00:14:57.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-26T00:14:57.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-26T00:14:57.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-26T00:14:58.263 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-26T00:14:58.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-26T00:14:58.840 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-26T00:14:58.859 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-26T00:14:58.859 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-26T00:14:58.859 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ11 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:14:59.129 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ11 hdr-ROBJ11 2024-03-26T00:14:59.354 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ11 key-ROBJ11 val-ROBJ11 2024-03-26T00:14:59.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-26T00:14:59.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ12 2024-03-26T00:14:59.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ12 2024-03-26T00:14:59.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-26T00:14:59.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-26T00:14:59.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ12 2024-03-26T00:14:59.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-26T00:14:59.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-26T00:14:59.516 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-26T00:14:59.968 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-26T00:14:59.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-26T00:15:00.462 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-26T00:15:00.483 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-26T00:15:00.483 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-26T00:15:00.483 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ12 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:15:00.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ12 hdr-ROBJ12 2024-03-26T00:15:01.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ12 key-ROBJ12 val-ROBJ12 2024-03-26T00:15:01.195 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-26T00:15:01.195 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ13 2024-03-26T00:15:01.195 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ13 2024-03-26T00:15:01.195 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-26T00:15:01.195 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-26T00:15:01.196 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ13 2024-03-26T00:15:01.196 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-26T00:15:01.196 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-26T00:15:01.196 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-26T00:15:01.657 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-26T00:15:01.674 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-26T00:15:02.177 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-26T00:15:02.198 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-26T00:15:02.198 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-26T00:15:02.198 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ13 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:15:02.525 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ13 hdr-ROBJ13 2024-03-26T00:15:02.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ13 key-ROBJ13 val-ROBJ13 2024-03-26T00:15:02.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-26T00:15:02.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ14 2024-03-26T00:15:02.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ14 2024-03-26T00:15:02.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-26T00:15:02.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-26T00:15:02.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ14 2024-03-26T00:15:02.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-26T00:15:02.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-26T00:15:02.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-26T00:15:03.406 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-26T00:15:03.425 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-26T00:15:03.983 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-26T00:15:04.006 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-26T00:15:04.006 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-26T00:15:04.006 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ14 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:15:04.298 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ14 hdr-ROBJ14 2024-03-26T00:15:04.663 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ14 key-ROBJ14 val-ROBJ14 2024-03-26T00:15:04.992 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-26T00:15:04.992 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ15 2024-03-26T00:15:04.993 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ15 2024-03-26T00:15:04.993 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-26T00:15:04.993 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-26T00:15:04.993 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ15 2024-03-26T00:15:04.993 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-26T00:15:04.993 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-26T00:15:04.994 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-26T00:15:05.479 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-26T00:15:05.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-26T00:15:06.014 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-26T00:15:06.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-26T00:15:06.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-26T00:15:06.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ15 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:15:06.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ15 hdr-ROBJ15 2024-03-26T00:15:06.462 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ15 key-ROBJ15 val-ROBJ15 2024-03-26T00:15:06.619 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-26T00:15:06.619 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ16 2024-03-26T00:15:06.619 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ16 2024-03-26T00:15:06.619 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-26T00:15:06.619 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-26T00:15:06.619 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ16 2024-03-26T00:15:06.619 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-26T00:15:06.619 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-26T00:15:06.619 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-26T00:15:07.068 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-26T00:15:07.089 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-26T00:15:07.561 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-26T00:15:07.582 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-26T00:15:07.582 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-26T00:15:07.582 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ16 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:15:07.851 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ16 hdr-ROBJ16 2024-03-26T00:15:08.031 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ16 key-ROBJ16 val-ROBJ16 2024-03-26T00:15:08.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-26T00:15:08.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ17 2024-03-26T00:15:08.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ17 2024-03-26T00:15:08.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-26T00:15:08.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-26T00:15:08.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ17 2024-03-26T00:15:08.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-26T00:15:08.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-26T00:15:08.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-26T00:15:08.690 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-26T00:15:08.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-26T00:15:09.141 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-26T00:15:09.161 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-26T00:15:09.161 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-26T00:15:09.161 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ17 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:15:09.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ17 hdr-ROBJ17 2024-03-26T00:15:09.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ17 key-ROBJ17 val-ROBJ17 2024-03-26T00:15:09.790 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-26T00:15:09.790 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ18 2024-03-26T00:15:09.791 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ18 2024-03-26T00:15:09.791 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-26T00:15:09.791 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-26T00:15:09.791 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ18 2024-03-26T00:15:09.791 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-26T00:15:09.791 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-26T00:15:09.791 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-26T00:15:10.237 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-26T00:15:10.253 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-26T00:15:10.923 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-26T00:15:10.951 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-26T00:15:10.951 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-26T00:15:10.951 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ18 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:15:11.270 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ18 hdr-ROBJ18 2024-03-26T00:15:11.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ18 key-ROBJ18 val-ROBJ18 2024-03-26T00:15:11.529 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-26T00:15:11.530 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ19 2024-03-26T00:15:11.530 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ19 2024-03-26T00:15:11.530 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-26T00:15:11.530 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-26T00:15:11.530 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ19 2024-03-26T00:15:11.530 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-26T00:15:11.530 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-26T00:15:11.530 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-26T00:15:11.968 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-26T00:15:11.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-26T00:15:12.461 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-26T00:15:12.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-26T00:15:12.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-26T00:15:12.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:15:12.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ19 hdr-ROBJ19 2024-03-26T00:15:13.152 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ19 key-ROBJ19 val-ROBJ19 2024-03-26T00:15:13.293 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1015: TEST_corrupt_scrub_replicated: dd if=/dev/zero of=td/osd-scrub-repair/new.ROBJ19 bs=1024 count=1025 2024-03-26T00:15:13.298 INFO:tasks.workunit.client.0.smithi129.stderr:1025+0 records in 2024-03-26T00:15:13.298 INFO:tasks.workunit.client.0.smithi129.stderr:1025+0 records out 2024-03-26T00:15:13.299 INFO:tasks.workunit.client.0.smithi129.stderr:1049600 bytes (1.0 MB, 1.0 MiB) copied, 0.00364443 s, 288 MB/s 2024-03-26T00:15:13.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/new.ROBJ19 2024-03-26T00:15:13.554 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1017: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ19 2024-03-26T00:15:13.557 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2024-03-26T00:15:13.557 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=csr_pool 2024-03-26T00:15:13.557 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=ROBJ0 2024-03-26T00:15:13.558 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-03-26T00:15:13.558 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-26T00:15:13.882 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: local pg=3.0 2024-03-26T00:15:13.883 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: get_primary csr_pool ROBJ0 2024-03-26T00:15:13.883 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=csr_pool 2024-03-26T00:15:13.883 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=ROBJ0 2024-03-26T00:15:13.883 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map csr_pool ROBJ0 2024-03-26T00:15:13.883 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-26T00:15:14.184 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: local primary=1 2024-03-26T00:15:14.184 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: get_asok_path osd.0 2024-03-26T00:15:14.185 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-26T00:15:14.185 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-26T00:15:14.185 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-26T00:15:14.185 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:15:14.185 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:15:14.185 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-osd.0.asok 2024-03-26T00:15:14.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2024-03-26T00:15:14.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.61636/ceph-osd.0.asok config set osd_deep_scrub_update_digest_min_age 0 2024-03-26T00:15:14.298 INFO:tasks.workunit.client.0.smithi129.stdout:{ 2024-03-26T00:15:14.298 INFO:tasks.workunit.client.0.smithi129.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2024-03-26T00:15:14.299 INFO:tasks.workunit.client.0.smithi129.stdout:} 2024-03-26T00:15:14.311 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: get_asok_path osd.1 2024-03-26T00:15:14.311 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-26T00:15:14.311 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-26T00:15:14.312 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-26T00:15:14.312 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:15:14.312 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:15:14.312 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-osd.1.asok 2024-03-26T00:15:14.313 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2024-03-26T00:15:14.313 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.61636/ceph-osd.1.asok config set osd_deep_scrub_update_digest_min_age 0 2024-03-26T00:15:14.416 INFO:tasks.workunit.client.0.smithi129.stdout:{ 2024-03-26T00:15:14.416 INFO:tasks.workunit.client.0.smithi129.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2024-03-26T00:15:14.416 INFO:tasks.workunit.client.0.smithi129.stdout:} 2024-03-26T00:15:14.430 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1027: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2024-03-26T00:15:14.430 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-03-26T00:15:14.431 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-26T00:15:14.431 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:15:14.432 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-26T00:15:14.432 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:15:14.432 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-26T00:15:14.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-03-26T00:14:34.528848+0000 2024-03-26T00:15:14.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-03-26T00:15:15.065 INFO:tasks.workunit.client.0.smithi129.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2024-03-26T00:15:15.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-03-26T00:14:34.528848+0000 last_deep_scrub_stamp 2024-03-26T00:15:15.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-26T00:15:15.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-26T00:14:34.528848+0000 2024-03-26T00:15:15.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-03-26T00:15:15.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-26T00:15:15.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:15:15.082 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-26T00:15:15.082 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:15:15.082 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-26T00:15:15.082 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:15:15.082 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-26T00:15:15.385 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:14:34.528848+0000 '>' 2024-03-26T00:14:34.528848+0000 2024-03-26T00:15:15.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:15:16.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:15:16.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:15:16.387 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-26T00:15:16.387 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:15:16.387 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-26T00:15:16.387 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:15:16.388 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-26T00:15:16.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:14:34.528848+0000 '>' 2024-03-26T00:14:34.528848+0000 2024-03-26T00:15:16.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:15:17.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:15:17.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:15:17.714 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-26T00:15:17.714 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:15:17.714 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-26T00:15:17.714 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:15:17.715 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-26T00:15:18.035 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:14:34.528848+0000 '>' 2024-03-26T00:14:34.528848+0000 2024-03-26T00:15:18.035 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:15:19.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:15:19.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:15:19.038 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-26T00:15:19.038 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:15:19.038 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-26T00:15:19.038 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:15:19.039 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-26T00:15:19.356 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:14:34.528848+0000 '>' 2024-03-26T00:14:34.528848+0000 2024-03-26T00:15:19.356 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:15:20.358 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:15:20.358 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:15:20.359 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-26T00:15:20.359 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:15:20.359 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-26T00:15:20.359 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:15:20.360 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-26T00:15:20.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:15:16.083704+0000 '>' 2024-03-26T00:14:34.528848+0000 2024-03-26T00:15:20.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-26T00:15:20.676 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: seq 1 19 2024-03-26T00:15:20.677 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-26T00:15:20.677 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ1 2024-03-26T00:15:20.678 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 1 % 2 2024-03-26T00:15:20.681 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-26T00:15:20.681 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-26T00:15:20.681 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1038: TEST_corrupt_scrub_replicated: local payload=UVWXYZZZ 2024-03-26T00:15:20.681 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1039: TEST_corrupt_scrub_replicated: echo UVWXYZZZ 2024-03-26T00:15:20.681 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1040: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-26T00:15:20.681 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:15:20.681 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:15:20.681 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-26T00:15:20.681 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:15:20.681 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-26T00:15:20.681 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:15:20.681 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:15:20.682 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-26T00:15:20.682 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:15:20.682 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-26T00:15:20.682 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:15:20.682 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:15:20.682 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:15:20.682 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:15:20.682 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:15:21.992 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:15:21.992 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-26T00:15:21.992 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:15:21.992 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:15:21.992 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:15:21.992 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:15:21.993 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:15:21.993 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-26T00:15:23.491 INFO:tasks.workunit.client.0.smithi129.stderr:Error getting attr on : 3.0_head,#-5:00000000:::scrub_3.0:head#, (61) No data available 2024-03-26T00:15:24.077 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-26T00:15:24.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:15:24.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:15:24.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-26T00:15:24.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:15:24.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:15:24.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:15:24.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:15:24.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:15:24.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:15:24.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:15:24.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:15:24.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:15:24.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:15:24.079 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:15:24.080 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:15:24.080 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:15:24.080 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:15:24.080 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:15:24.080 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:15:24.081 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:15:24.081 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:15:24.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:15:24.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:15:24.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:15:24.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:15:24.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:15:24.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:15:24.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:15:24.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:15:24.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:15:24.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:15:24.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:15:24.082 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-26T00:15:24.082 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.1 2024-03-26T00:15:24.082 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:15:24.083 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-26T00:15:24.083 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-26T00:15:24.084 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:15:24.084 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:15:24.084 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:15:24.121 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:15:24.118+0000 7f3f468aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:15:24.123 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:15:24.122+0000 7f3f468aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:15:24.125 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:15:24.122+0000 7f3f468aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:15:24.400 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-26T00:15:24.400 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:15:24.400 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:15:24.400 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:15:24.400 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:15:24.400 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:15:24.400 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:15:24.400 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:15:24.401 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:15:24.401 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:15:24.428 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:15:24.426+0000 7f3f468aa7c0 -1 Falling back to public interface 2024-03-26T00:15:24.799 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:15:25.801 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:15:25.801 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:15:25.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:15:25.802 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:15:25.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:15:25.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:15:26.118 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:15:27.120 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:15:27.120 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:15:27.120 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:15:27.120 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:15:27.120 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:15:27.120 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:15:27.297 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:15:27.294+0000 7f3f468aa7c0 -1 osd.1 63 log_to_monitors true 2024-03-26T00:15:27.446 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:15:28.448 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:15:28.448 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:15:28.449 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:15:28.449 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:15:28.449 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:15:28.449 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:15:28.918 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1 up in weight 1 up_from 67 up_thru 67 down_at 64 last_clean_interval [10,63) [v2:127.0.0.1:6810/350524875,v1:127.0.0.1:6811/350524875] [v2:127.0.0.1:6812/350524875,v1:127.0.0.1:6813/350524875] exists,up 0c2624cc-3bc6-4489-918a-a2155ff79e95 2024-03-26T00:15:28.918 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:15:28.918 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:15:28.918 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:15:28.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:15:28.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:15:28.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:15:28.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:15:28.919 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:15:28.920 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:15:28.920 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:15:28.920 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:15:28.920 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:15:28.920 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:15:29.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:15:29.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:15:29.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:15:29.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:15:29.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:15:29.036 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:15:29.366 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:15:29.366 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:15:29.366 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:15:29.366 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:15:29.367 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:15:29.512 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836509 2024-03-26T00:15:29.512 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836509 2024-03-26T00:15:29.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509' 2024-03-26T00:15:29.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:15:29.513 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:15:29.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808834 2024-03-26T00:15:29.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808834 2024-03-26T00:15:29.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-287762808834' 2024-03-26T00:15:29.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:15:29.645 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836509 2024-03-26T00:15:29.645 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:15:29.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:15:29.647 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836509 2024-03-26T00:15:29.647 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:15:29.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836509 2024-03-26T00:15:29.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-03-26T00:15:29.649 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 21474836509 2024-03-26T00:15:29.649 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:15:29.950 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836509 2024-03-26T00:15:29.951 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:15:30.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:15:30.953 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:15:31.259 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836509 -lt 21474836509 2024-03-26T00:15:31.259 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:15:31.261 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-287762808834 2024-03-26T00:15:31.261 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:15:31.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:15:31.264 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-287762808834 2024-03-26T00:15:31.265 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:15:31.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808834 2024-03-26T00:15:31.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 287762808834' 2024-03-26T00:15:31.267 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 287762808834 2024-03-26T00:15:31.267 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:15:31.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808834 -lt 287762808834 2024-03-26T00:15:31.608 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:15:31.608 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:15:31.609 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:15:31.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:15:31.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:15:31.973 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:15:31.973 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:15:31.973 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:15:31.973 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:15:31.973 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:15:31.974 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:15:32.253 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:15:32.254 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:15:32.254 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:15:32.254 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:15:32.629 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:15:32.629 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:15:32.629 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:15:32.629 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-26T00:15:32.629 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ2 2024-03-26T00:15:32.629 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 2 % 2 2024-03-26T00:15:32.629 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-26T00:15:32.629 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-26T00:15:32.629 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1045: TEST_corrupt_scrub_replicated: local payload=UVWXYZ 2024-03-26T00:15:32.629 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: echo UVWXYZ 2024-03-26T00:15:32.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1047: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-26T00:15:32.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:15:32.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:15:32.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-26T00:15:32.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:15:32.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-26T00:15:32.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:15:32.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:15:32.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-26T00:15:32.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:15:32.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-26T00:15:32.630 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:15:32.630 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:15:32.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:15:32.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:15:32.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:15:33.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:15:33.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-26T00:15:33.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:15:33.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:15:33.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:15:33.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:15:33.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:15:33.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-26T00:15:36.270 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-26T00:15:36.271 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:15:36.271 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:15:36.271 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-26T00:15:36.271 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:15:36.271 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:15:36.271 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:15:36.271 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:15:36.271 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:15:36.271 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:15:36.271 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-26T00:15:36.271 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-26T00:15:36.272 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:15:36.272 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:15:36.272 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:15:36.272 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:15:36.272 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:15:36.272 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:15:36.273 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:15:36.273 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:15:36.273 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:15:36.274 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:15:36.274 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:15:36.274 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:15:36.274 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:15:36.274 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:15:36.274 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:15:36.274 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:15:36.274 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:15:36.274 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:15:36.274 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:15:36.274 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:15:36.274 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-26T00:15:36.275 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-26T00:15:36.275 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.0 2024-03-26T00:15:36.275 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:15:36.276 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-26T00:15:36.277 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-26T00:15:36.277 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:15:36.277 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:15:36.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:15:36.309 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:15:36.310+0000 7fcc24e7e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:15:36.312 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:15:36.314+0000 7fcc24e7e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:15:36.313 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:15:36.314+0000 7fcc24e7e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:15:36.592 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-26T00:15:36.592 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:15:36.592 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-26T00:15:36.592 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:15:36.593 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:15:36.593 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:15:36.593 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:15:36.593 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:15:36.593 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:15:36.593 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:15:36.848 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:15:36.850+0000 7fcc24e7e7c0 -1 Falling back to public interface 2024-03-26T00:15:36.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:15:37.894 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:15:37.894 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:15:37.894 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:15:37.894 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:15:37.895 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:15:37.895 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:15:38.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:15:39.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:15:39.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:15:39.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:15:39.199 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:15:39.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:15:39.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:15:39.415 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:15:39.418+0000 7fcc24e7e7c0 -1 osd.0 68 log_to_monitors true 2024-03-26T00:15:39.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:15:40.548 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:15:40.548 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:15:40.548 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:15:40.548 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:15:40.549 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:15:40.549 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:15:41.130 INFO:tasks.workunit.client.0.smithi129.stderr:osd.0 up in weight 1 up_from 72 up_thru 72 down_at 69 last_clean_interval [5,68) [v2:127.0.0.1:6802/3249513571,v1:127.0.0.1:6803/3249513571] [v2:127.0.0.1:6804/3249513571,v1:127.0.0.1:6805/3249513571] exists,up d7a8ca59-f9ab-4120-b1c3-f87ea4b0bd36 2024-03-26T00:15:41.130 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:15:41.130 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:15:41.130 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:15:41.130 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:15:41.130 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:15:41.131 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:15:41.131 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:15:41.131 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:15:41.131 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:15:41.131 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:15:41.132 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:15:41.132 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:15:41.132 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:15:41.259 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:15:41.259 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:15:41.259 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:15:41.259 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:15:41.259 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:15:41.259 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:15:41.574 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:15:41.575 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:15:41.575 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:15:41.575 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:15:41.575 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:15:41.722 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645314 2024-03-26T00:15:41.722 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645314 2024-03-26T00:15:41.722 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-309237645314' 2024-03-26T00:15:41.722 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:15:41.723 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:15:41.864 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808837 2024-03-26T00:15:41.864 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808837 2024-03-26T00:15:41.864 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-309237645314 1-287762808837' 2024-03-26T00:15:41.864 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:15:41.865 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:15:41.865 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-309237645314 2024-03-26T00:15:41.866 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:15:41.867 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-309237645314 2024-03-26T00:15:41.867 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:15:41.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645314 2024-03-26T00:15:41.868 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 309237645314' 2024-03-26T00:15:41.868 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 309237645314 2024-03-26T00:15:41.868 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:15:42.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 309237645314 2024-03-26T00:15:42.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:15:43.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:15:43.177 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:15:43.488 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645314 -lt 309237645314 2024-03-26T00:15:43.488 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:15:43.489 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-287762808837 2024-03-26T00:15:43.489 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:15:43.491 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:15:43.492 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-287762808837 2024-03-26T00:15:43.492 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:15:43.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808837 2024-03-26T00:15:43.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 287762808837' 2024-03-26T00:15:43.494 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 287762808837 2024-03-26T00:15:43.494 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:15:43.859 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808837 -lt 287762808837 2024-03-26T00:15:43.860 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:15:43.861 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:15:43.861 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:15:44.242 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:15:44.242 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:15:44.243 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:15:44.243 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:15:44.243 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:15:44.243 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:15:44.243 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:15:44.243 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:15:44.526 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:15:44.527 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:15:44.527 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:15:44.527 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:15:44.881 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:15:44.881 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:15:44.881 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:15:44.881 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-26T00:15:44.881 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ3 2024-03-26T00:15:44.881 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 3 % 2 2024-03-26T00:15:44.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-26T00:15:44.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-26T00:15:44.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1052: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ3 remove 2024-03-26T00:15:44.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:15:44.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:15:44.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-26T00:15:44.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:15:44.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ3 remove 2024-03-26T00:15:44.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:15:44.885 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:15:44.885 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-26T00:15:44.885 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:15:44.885 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-26T00:15:44.885 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:15:44.885 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:15:44.885 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:15:44.885 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:15:44.885 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:15:46.193 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:15:46.193 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ3 remove 2024-03-26T00:15:46.193 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:15:46.193 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:15:46.193 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:15:46.193 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:15:46.193 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:15:46.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ3 remove 2024-03-26T00:15:47.581 INFO:tasks.workunit.client.0.smithi129.stdout:remove #3:f2a5b2a4:::ROBJ3:head# 2024-03-26T00:15:48.118 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-26T00:15:48.118 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:15:48.118 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:15:48.118 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-26T00:15:48.118 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:15:48.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:15:48.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:15:48.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:15:48.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:15:48.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:15:48.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:15:48.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:15:48.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:15:48.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:15:48.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:15:48.121 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:15:48.121 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:15:48.121 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:15:48.121 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:15:48.121 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:15:48.121 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:15:48.121 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:15:48.121 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:15:48.121 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:15:48.121 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:15:48.121 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:15:48.121 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:15:48.122 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:15:48.122 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:15:48.122 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:15:48.122 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:15:48.122 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:15:48.122 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:15:48.122 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-26T00:15:48.123 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.1 2024-03-26T00:15:48.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:15:48.123 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-26T00:15:48.125 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-26T00:15:48.125 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:15:48.126 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:15:48.126 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:15:48.157 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:15:48.157+0000 7f54354a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:15:48.159 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:15:48.161+0000 7f54354a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:15:48.161 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:15:48.161+0000 7f54354a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:15:48.432 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-26T00:15:48.432 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:15:48.432 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:15:48.432 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:15:48.432 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:15:48.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:15:48.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:15:48.433 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:15:48.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:15:48.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:15:48.693 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:15:48.693+0000 7f54354a87c0 -1 Falling back to public interface 2024-03-26T00:15:48.748 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:15:49.749 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:15:49.749 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:15:49.749 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:15:49.749 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:15:49.750 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:15:49.750 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:15:50.056 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:15:51.058 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:15:51.058 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:15:51.058 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:15:51.058 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:15:51.059 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:15:51.059 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:15:51.424 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:15:51.425+0000 7f54354a87c0 -1 osd.1 73 log_to_monitors true 2024-03-26T00:15:51.439 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:15:52.440 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:15:52.441 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:15:52.441 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:15:52.441 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:15:52.442 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:15:52.442 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:15:52.908 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1 up in weight 1 up_from 77 up_thru 77 down_at 74 last_clean_interval [67,73) [v2:127.0.0.1:6810/419520124,v1:127.0.0.1:6811/419520124] [v2:127.0.0.1:6812/419520124,v1:127.0.0.1:6813/419520124] exists,up 0c2624cc-3bc6-4489-918a-a2155ff79e95 2024-03-26T00:15:52.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:15:52.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:15:52.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:15:52.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:15:52.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:15:52.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:15:52.909 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:15:52.909 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:15:52.909 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:15:52.909 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:15:52.910 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:15:52.910 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:15:52.910 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:15:53.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:15:53.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:15:53.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:15:53.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:15:53.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:15:53.027 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:15:53.356 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:15:53.356 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:15:53.357 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:15:53.357 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:15:53.357 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:15:53.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645317 2024-03-26T00:15:53.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645317 2024-03-26T00:15:53.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-309237645317' 2024-03-26T00:15:53.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:15:53.547 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:15:53.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481794 2024-03-26T00:15:53.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481794 2024-03-26T00:15:53.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-309237645317 1-330712481794' 2024-03-26T00:15:53.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:15:53.689 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-309237645317 2024-03-26T00:15:53.689 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:15:53.690 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:15:53.691 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-309237645317 2024-03-26T00:15:53.691 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:15:53.692 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645317 2024-03-26T00:15:53.692 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 309237645317' 2024-03-26T00:15:53.692 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 309237645317 2024-03-26T00:15:53.692 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:15:53.996 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645316 -lt 309237645317 2024-03-26T00:15:53.996 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:15:54.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:15:54.999 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:15:55.324 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645317 -lt 309237645317 2024-03-26T00:15:55.324 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:15:55.325 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-330712481794 2024-03-26T00:15:55.325 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:15:55.327 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:15:55.328 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-330712481794 2024-03-26T00:15:55.328 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:15:55.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481794 2024-03-26T00:15:55.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 330712481794' 2024-03-26T00:15:55.330 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 330712481794 2024-03-26T00:15:55.330 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:15:55.677 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481794 -lt 330712481794 2024-03-26T00:15:55.678 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:15:55.679 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:15:55.679 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:15:56.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:15:56.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:15:56.043 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:15:56.043 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:15:56.043 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:15:56.043 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:15:56.043 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:15:56.043 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:15:56.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:15:56.331 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:15:56.331 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:15:56.332 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:15:56.682 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:15:56.682 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:15:56.683 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:15:56.683 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-26T00:15:56.683 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ4 2024-03-26T00:15:56.683 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 4 % 2 2024-03-26T00:15:56.684 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-26T00:15:56.684 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-26T00:15:56.685 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1057: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-03-26T00:15:56.685 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:15:56.685 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:15:56.685 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-26T00:15:56.685 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:15:56.685 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-03-26T00:15:56.685 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:15:56.685 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:15:56.685 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-26T00:15:56.685 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:15:56.685 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-26T00:15:56.685 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:15:56.685 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:15:56.685 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:15:56.686 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:15:56.686 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:15:56.995 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:15:56.995 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-03-26T00:15:56.995 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:15:56.995 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:15:56.995 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:15:56.995 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:15:56.995 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:15:56.995 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-03-26T00:15:58.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-26T00:15:58.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:15:58.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:15:58.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-26T00:15:58.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:15:58.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:15:58.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:15:58.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:15:58.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:15:58.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:15:58.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-26T00:15:58.999 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-26T00:15:58.999 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:15:58.999 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:15:58.999 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:15:59.000 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:15:59.000 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:15:59.000 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:15:59.000 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:15:59.000 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:15:59.001 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:15:59.001 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:15:59.001 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:15:59.001 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:15:59.001 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:15:59.001 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:15:59.001 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:15:59.001 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:15:59.001 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:15:59.001 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:15:59.001 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:15:59.001 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:15:59.001 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-26T00:15:59.002 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-26T00:15:59.002 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.0 2024-03-26T00:15:59.003 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:15:59.003 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-26T00:15:59.005 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-26T00:15:59.005 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:15:59.005 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:15:59.005 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:15:59.037 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:15:59.037+0000 7f673d9fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:15:59.054 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:15:59.053+0000 7f673d9fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:15:59.055 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:15:59.057+0000 7f673d9fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:15:59.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-26T00:15:59.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:15:59.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-26T00:15:59.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:15:59.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:15:59.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:15:59.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:15:59.348 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:15:59.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:15:59.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:15:59.589 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:15:59.589+0000 7f673d9fb7c0 -1 Falling back to public interface 2024-03-26T00:15:59.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:16:00.728 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:16:00.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:16:00.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:16:00.729 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:16:00.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:16:00.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:16:01.063 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:16:02.064 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:16:02.065 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:16:02.065 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:16:02.065 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:16:02.065 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:16:02.065 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:16:02.170 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:16:02.169+0000 7f673d9fb7c0 -1 osd.0 78 log_to_monitors true 2024-03-26T00:16:02.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:16:03.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:16:03.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:16:03.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:16:03.459 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:16:03.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:16:03.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:16:03.921 INFO:tasks.workunit.client.0.smithi129.stderr:osd.0 up in weight 1 up_from 82 up_thru 82 down_at 79 last_clean_interval [72,78) [v2:127.0.0.1:6802/1867537593,v1:127.0.0.1:6803/1867537593] [v2:127.0.0.1:6804/1867537593,v1:127.0.0.1:6805/1867537593] exists,up d7a8ca59-f9ab-4120-b1c3-f87ea4b0bd36 2024-03-26T00:16:03.921 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:16:03.921 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:16:03.921 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:16:03.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:16:03.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:16:03.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:16:03.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:16:03.923 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:16:03.923 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:16:03.924 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:16:03.924 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:16:03.924 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:16:03.924 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:16:04.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:16:04.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:16:04.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:16:04.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:16:04.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:16:04.045 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:16:04.405 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:16:04.405 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:16:04.405 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:16:04.405 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:16:04.405 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:16:04.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318274 2024-03-26T00:16:04.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318274 2024-03-26T00:16:04.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-352187318274' 2024-03-26T00:16:04.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:16:04.630 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:16:04.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481797 2024-03-26T00:16:04.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481797 2024-03-26T00:16:04.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-352187318274 1-330712481797' 2024-03-26T00:16:04.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:16:04.780 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-352187318274 2024-03-26T00:16:04.780 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:16:04.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:16:04.782 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-352187318274 2024-03-26T00:16:04.782 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:16:04.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318274 2024-03-26T00:16:04.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 352187318274' 2024-03-26T00:16:04.784 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 352187318274 2024-03-26T00:16:04.784 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:16:05.088 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 352187318274 2024-03-26T00:16:05.088 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:16:06.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:16:06.091 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:16:06.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 352187318274 2024-03-26T00:16:06.416 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:16:07.416 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-26T00:16:07.417 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:16:07.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318274 -lt 352187318274 2024-03-26T00:16:07.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:16:07.733 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-330712481797 2024-03-26T00:16:07.733 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:16:07.738 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:16:07.739 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-330712481797 2024-03-26T00:16:07.739 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:16:07.739 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481797 2024-03-26T00:16:07.739 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 330712481797' 2024-03-26T00:16:07.739 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 330712481797 2024-03-26T00:16:07.739 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:16:08.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481797 -lt 330712481797 2024-03-26T00:16:08.095 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:16:08.096 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:16:08.096 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:16:08.462 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:16:08.462 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:16:08.463 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:16:08.463 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:16:08.463 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:16:08.463 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:16:08.463 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:16:08.463 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:16:08.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:16:08.758 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:16:08.758 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:16:08.759 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:16:09.107 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:16:09.107 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:16:09.107 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:16:09.107 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-26T00:16:09.107 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ5 2024-03-26T00:16:09.107 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 5 % 2 2024-03-26T00:16:09.109 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-26T00:16:09.109 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-26T00:16:09.110 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1062: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-03-26T00:16:09.110 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:16:09.110 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:16:09.110 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-26T00:16:09.110 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:16:09.110 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-03-26T00:16:09.110 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:16:09.110 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:16:09.110 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-26T00:16:09.110 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:16:09.110 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-26T00:16:09.111 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:16:09.111 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:16:09.111 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:16:09.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:16:09.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:16:09.421 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:16:09.422 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-03-26T00:16:09.422 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:16:09.422 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:16:09.422 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:16:09.422 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:16:09.422 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:16:09.422 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ5 rm-omap key-ROBJ5 2024-03-26T00:16:11.601 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-26T00:16:11.601 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:16:11.601 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:16:11.601 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-26T00:16:11.601 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:16:11.601 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:16:11.601 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:16:11.601 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:16:11.601 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:16:11.601 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:16:11.602 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:16:11.602 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:16:11.602 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:16:11.602 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:16:11.602 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:16:11.603 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:16:11.603 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:16:11.603 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:16:11.604 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:16:11.604 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:16:11.604 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:16:11.604 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:16:11.604 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:16:11.604 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:16:11.604 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:16:11.604 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:16:11.604 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:16:11.604 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:16:11.604 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:16:11.604 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:16:11.604 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:16:11.604 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:16:11.605 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:16:11.605 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-26T00:16:11.605 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.1 2024-03-26T00:16:11.605 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:16:11.605 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-26T00:16:11.606 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-26T00:16:11.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:16:11.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:16:11.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:16:11.640 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:16:11.641+0000 7f1f51d3c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:16:11.642 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:16:11.641+0000 7f1f51d3c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:16:11.644 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:16:11.645+0000 7f1f51d3c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:16:11.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-26T00:16:11.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:16:11.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:16:11.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:16:11.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:16:11.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:16:11.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:16:11.927 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:16:11.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:16:11.928 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:16:12.177 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:16:12.177+0000 7f1f51d3c7c0 -1 Falling back to public interface 2024-03-26T00:16:12.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:16:13.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:16:13.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:16:13.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:16:13.227 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:16:13.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:16:13.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:16:13.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:16:14.530 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:16:14.530 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:16:14.530 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:16:14.530 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:16:14.531 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:16:14.531 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:16:14.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:16:14.894 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:16:14.893+0000 7f1f51d3c7c0 -1 osd.1 84 log_to_monitors true 2024-03-26T00:16:15.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:16:15.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:16:15.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:16:15.838 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:16:15.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:16:15.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:16:16.219 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1 up in weight 1 up_from 88 up_thru 83 down_at 85 last_clean_interval [77,84) [v2:127.0.0.1:6810/2629454704,v1:127.0.0.1:6811/2629454704] [v2:127.0.0.1:6812/2629454704,v1:127.0.0.1:6813/2629454704] exists,up 0c2624cc-3bc6-4489-918a-a2155ff79e95 2024-03-26T00:16:16.219 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:16:16.219 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:16:16.219 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:16:16.219 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:16:16.219 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:16:16.219 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:16:16.219 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:16:16.220 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:16:16.220 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:16:16.220 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:16:16.220 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:16:16.220 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:16:16.220 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:16:16.335 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:16:16.335 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:16:16.335 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:16:16.335 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:16:16.335 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:16:16.336 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:16:16.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:16:16.779 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:16:16.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:16:16.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:16:16.780 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:16:17.068 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318277 2024-03-26T00:16:17.068 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318277 2024-03-26T00:16:17.068 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-352187318277' 2024-03-26T00:16:17.068 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:16:17.068 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:16:17.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122050 2024-03-26T00:16:17.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122050 2024-03-26T00:16:17.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-352187318277 1-377957122050' 2024-03-26T00:16:17.255 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:16:17.255 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:16:17.255 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-352187318277 2024-03-26T00:16:17.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:16:17.257 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-352187318277 2024-03-26T00:16:17.257 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:16:17.258 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318277 2024-03-26T00:16:17.258 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 352187318277' 2024-03-26T00:16:17.258 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 352187318277 2024-03-26T00:16:17.258 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:16:17.571 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318276 -lt 352187318277 2024-03-26T00:16:17.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:16:18.573 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:16:18.574 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:16:18.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318278 -lt 352187318277 2024-03-26T00:16:18.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:16:18.898 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-377957122050 2024-03-26T00:16:18.898 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:16:18.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:16:18.899 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-377957122050 2024-03-26T00:16:18.900 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:16:18.901 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122050 2024-03-26T00:16:18.901 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 377957122050' 2024-03-26T00:16:18.901 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 377957122050 2024-03-26T00:16:18.902 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:16:19.255 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122050 -lt 377957122050 2024-03-26T00:16:19.256 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:16:19.256 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:16:19.256 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:16:19.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:16:19.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:16:19.610 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:16:19.610 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:16:19.610 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:16:19.611 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:16:19.611 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:16:19.611 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:16:19.921 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:16:19.922 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:16:19.922 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:16:19.922 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:16:20.305 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:16:20.305 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:16:20.305 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:16:20.305 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-26T00:16:20.305 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ6 2024-03-26T00:16:20.305 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 6 % 2 2024-03-26T00:16:20.308 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-26T00:16:20.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-26T00:16:20.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: echo extra 2024-03-26T00:16:20.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1068: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-03-26T00:16:20.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:16:20.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:16:20.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-26T00:16:20.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:16:20.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-03-26T00:16:20.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:16:20.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:16:20.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-26T00:16:20.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:16:20.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-26T00:16:20.310 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:16:20.310 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:16:20.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:16:20.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:16:20.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:16:21.620 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:16:21.620 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-03-26T00:16:21.620 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:16:21.620 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:16:21.620 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:16:21.620 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:16:21.620 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:16:21.621 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-03-26T00:16:23.412 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-26T00:16:23.412 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:16:23.412 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:16:23.412 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-26T00:16:23.412 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:16:23.412 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:16:23.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:16:23.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:16:23.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:16:23.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:16:23.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-26T00:16:23.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-26T00:16:23.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:16:23.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:16:23.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:16:23.413 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:16:23.414 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:16:23.414 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:16:23.418 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:16:23.418 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:16:23.418 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:16:23.418 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:16:23.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:16:23.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:16:23.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:16:23.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:16:23.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:16:23.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:16:23.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:16:23.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:16:23.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:16:23.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:16:23.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-26T00:16:23.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-26T00:16:23.419 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.0 2024-03-26T00:16:23.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:16:23.419 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-26T00:16:23.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-26T00:16:23.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:16:23.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:16:23.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:16:23.451 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:16:23.449+0000 7f64a92e87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:16:23.453 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:16:23.453+0000 7f64a92e87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:16:23.454 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:16:23.453+0000 7f64a92e87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:16:23.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-26T00:16:23.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:16:23.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-26T00:16:23.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:16:23.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:16:23.744 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:16:23.744 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:16:23.744 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:16:23.744 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:16:23.744 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:16:24.006 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:16:24.005+0000 7f64a92e87c0 -1 Falling back to public interface 2024-03-26T00:16:24.059 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:16:25.060 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:16:25.061 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:16:25.061 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:16:25.061 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:16:25.061 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:16:25.061 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:16:25.384 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:16:26.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:16:26.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:16:26.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:16:26.386 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:16:26.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:16:26.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:16:26.640 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:16:26.641+0000 7f64a92e87c0 -1 osd.0 89 log_to_monitors true 2024-03-26T00:16:26.788 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:16:27.790 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:16:27.790 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:16:27.790 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:16:27.790 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:16:27.791 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:16:27.791 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:16:28.269 INFO:tasks.workunit.client.0.smithi129.stderr:osd.0 up in weight 1 up_from 93 up_thru 93 down_at 90 last_clean_interval [82,89) [v2:127.0.0.1:6802/1300157916,v1:127.0.0.1:6803/1300157916] [v2:127.0.0.1:6804/1300157916,v1:127.0.0.1:6805/1300157916] exists,up d7a8ca59-f9ab-4120-b1c3-f87ea4b0bd36 2024-03-26T00:16:28.289 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:16:28.290 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:16:28.290 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:16:28.290 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:16:28.290 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:16:28.290 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:16:28.290 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:16:28.290 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:16:28.291 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:16:28.291 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:16:28.291 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:16:28.291 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:16:28.291 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:16:28.385 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:16:28.385 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:16:28.385 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:16:28.385 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:16:28.385 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:16:28.385 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:16:28.724 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:16:28.724 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:16:28.725 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:16:28.725 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:16:28.725 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:16:28.986 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=399431958530 2024-03-26T00:16:28.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 399431958530 2024-03-26T00:16:28.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-399431958530' 2024-03-26T00:16:28.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:16:28.987 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:16:29.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122053 2024-03-26T00:16:29.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122053 2024-03-26T00:16:29.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-399431958530 1-377957122053' 2024-03-26T00:16:29.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:16:29.182 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-399431958530 2024-03-26T00:16:29.183 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:16:29.183 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:16:29.184 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-399431958530 2024-03-26T00:16:29.184 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:16:29.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=399431958530 2024-03-26T00:16:29.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 399431958530' 2024-03-26T00:16:29.186 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 399431958530 2024-03-26T00:16:29.186 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:16:29.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 399431958530 2024-03-26T00:16:29.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:16:30.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:16:30.507 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:16:30.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958530 -lt 399431958530 2024-03-26T00:16:30.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:16:30.831 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-377957122053 2024-03-26T00:16:30.832 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:16:30.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:16:30.834 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-377957122053 2024-03-26T00:16:30.834 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:16:30.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122053 2024-03-26T00:16:30.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 377957122053' 2024-03-26T00:16:30.836 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 377957122053 2024-03-26T00:16:30.836 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:16:31.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122054 -lt 377957122053 2024-03-26T00:16:31.143 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:16:31.143 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:16:31.143 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:16:31.488 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:16:31.488 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:16:31.489 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:16:31.489 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:16:31.489 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:16:31.489 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:16:31.490 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:16:31.490 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:16:31.794 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:16:31.794 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:16:31.795 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:16:31.795 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:16:32.161 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:16:32.161 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:16:32.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:16:32.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/extra-val 2024-03-26T00:16:32.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-26T00:16:32.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ7 2024-03-26T00:16:32.163 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 7 % 2 2024-03-26T00:16:32.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-26T00:16:32.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-26T00:16:32.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1074: TEST_corrupt_scrub_replicated: echo -n newheader 2024-03-26T00:16:32.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1075: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-03-26T00:16:32.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:16:32.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:16:32.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-26T00:16:32.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:16:32.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-03-26T00:16:32.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:16:32.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:16:32.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-26T00:16:32.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:16:32.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-26T00:16:32.166 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:16:32.166 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:16:32.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:16:32.167 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:16:32.167 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:16:33.478 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:16:33.478 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-03-26T00:16:33.479 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:16:33.479 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:16:33.479 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:16:33.479 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:16:33.479 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:16:33.479 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-03-26T00:16:34.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-26T00:16:34.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:16:34.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:16:34.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-26T00:16:34.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:16:34.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:16:34.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:16:34.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:16:34.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:16:34.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:16:34.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:16:34.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:16:34.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:16:34.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:16:34.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:16:34.988 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:16:34.988 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:16:34.988 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:16:34.989 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:16:34.989 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:16:34.989 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:16:34.989 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:16:34.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:16:34.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:16:34.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:16:34.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:16:34.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:16:34.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:16:34.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:16:34.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:16:34.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:16:34.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:16:34.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:16:34.992 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-26T00:16:34.992 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.1 2024-03-26T00:16:34.992 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:16:34.992 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-26T00:16:34.993 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-26T00:16:34.994 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:16:34.994 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:16:34.994 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:16:35.026 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:16:35.024+0000 7fa34be057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:16:35.050 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:16:35.048+0000 7fa34be057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:16:35.051 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:16:35.052+0000 7fa34be057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:16:35.340 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-26T00:16:35.340 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:16:35.340 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:16:35.340 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:16:35.340 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:16:35.340 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:16:35.340 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:16:35.340 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:16:35.340 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:16:35.340 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:16:35.586 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:16:35.584+0000 7fa34be057c0 -1 Falling back to public interface 2024-03-26T00:16:35.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:16:36.709 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:16:36.709 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:16:36.709 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:16:36.709 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:16:36.709 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:16:36.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:16:37.019 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:16:38.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:16:38.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:16:38.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:16:38.022 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:16:38.022 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:16:38.022 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:16:38.163 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:16:38.164+0000 7fa34be057c0 -1 osd.1 95 log_to_monitors true 2024-03-26T00:16:38.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:16:39.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:16:39.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:16:39.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:16:39.344 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:16:39.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:16:39.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:16:39.897 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1 up in weight 1 up_from 99 up_thru 99 down_at 96 last_clean_interval [88,95) [v2:127.0.0.1:6810/1004397599,v1:127.0.0.1:6811/1004397599] [v2:127.0.0.1:6812/1004397599,v1:127.0.0.1:6813/1004397599] exists,up 0c2624cc-3bc6-4489-918a-a2155ff79e95 2024-03-26T00:16:39.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:16:39.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:16:39.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:16:39.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:16:39.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:16:39.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:16:39.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:16:39.898 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:16:39.899 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:16:39.899 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:16:39.899 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:16:39.899 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:16:39.899 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:16:40.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:16:40.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:16:40.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:16:40.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:16:40.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:16:40.021 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:16:40.401 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:16:40.401 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:16:40.402 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:16:40.402 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:16:40.402 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:16:40.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=399431958533 2024-03-26T00:16:40.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 399431958533 2024-03-26T00:16:40.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-399431958533' 2024-03-26T00:16:40.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:16:40.627 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:16:40.769 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=425201762306 2024-03-26T00:16:40.769 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 425201762306 2024-03-26T00:16:40.769 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-399431958533 1-425201762306' 2024-03-26T00:16:40.769 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:16:40.769 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-399431958533 2024-03-26T00:16:40.770 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:16:40.771 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:16:40.771 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-399431958533 2024-03-26T00:16:40.771 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:16:40.773 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=399431958533 2024-03-26T00:16:40.773 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 399431958533' 2024-03-26T00:16:40.773 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 399431958533 2024-03-26T00:16:40.773 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:16:41.072 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958532 -lt 399431958533 2024-03-26T00:16:41.073 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:16:42.074 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:16:42.074 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:16:42.380 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958532 -lt 399431958533 2024-03-26T00:16:42.380 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:16:43.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-26T00:16:43.383 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:16:43.701 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958534 -lt 399431958533 2024-03-26T00:16:43.701 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:16:43.702 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-425201762306 2024-03-26T00:16:43.702 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:16:43.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:16:43.704 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-425201762306 2024-03-26T00:16:43.705 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:16:43.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=425201762306 2024-03-26T00:16:43.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 425201762306' 2024-03-26T00:16:43.706 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 425201762306 2024-03-26T00:16:43.706 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:16:44.022 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 425201762306 -lt 425201762306 2024-03-26T00:16:44.023 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:16:44.023 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:16:44.023 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:16:44.397 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:16:44.398 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:16:44.399 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:16:44.399 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:16:44.399 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:16:44.399 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:16:44.399 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:16:44.399 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:16:44.692 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:16:44.693 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:16:44.693 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:16:44.693 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:16:45.075 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:16:45.075 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:16:45.075 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:16:45.075 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1076: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/hdr 2024-03-26T00:16:45.076 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-26T00:16:45.076 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ8 2024-03-26T00:16:45.076 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 8 % 2 2024-03-26T00:16:45.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-26T00:16:45.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-26T00:16:45.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1080: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key1-ROBJ8 val1-ROBJ8 2024-03-26T00:16:45.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1081: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key2-ROBJ8 val2-ROBJ8 2024-03-26T00:16:45.372 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1084: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-03-26T00:16:45.372 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1085: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-03-26T00:16:45.372 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:16:45.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:16:45.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-26T00:16:45.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:16:45.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-03-26T00:16:45.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:16:45.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:16:45.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-26T00:16:45.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:16:45.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-26T00:16:45.373 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:16:45.373 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:16:45.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:16:45.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:16:45.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:16:45.683 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:16:45.683 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-03-26T00:16:45.683 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:16:45.683 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:16:45.683 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:16:45.683 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:16:45.683 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:16:45.683 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-03-26T00:16:47.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-26T00:16:47.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:16:47.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:16:47.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-26T00:16:47.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:16:47.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:16:47.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:16:47.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:16:47.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:16:47.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:16:47.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-26T00:16:47.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-26T00:16:47.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:16:47.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:16:47.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:16:47.632 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:16:47.633 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:16:47.633 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:16:47.633 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:16:47.633 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:16:47.633 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:16:47.633 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:16:47.633 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:16:47.633 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:16:47.633 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:16:47.633 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:16:47.633 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:16:47.633 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:16:47.633 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:16:47.633 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:16:47.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:16:47.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:16:47.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-26T00:16:47.635 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-26T00:16:47.635 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.0 2024-03-26T00:16:47.635 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:16:47.636 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-26T00:16:47.637 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-26T00:16:47.638 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:16:47.638 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:16:47.638 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:16:47.668 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:16:47.668+0000 7f2f11bec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:16:47.671 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:16:47.672+0000 7f2f11bec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:16:47.673 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:16:47.672+0000 7f2f11bec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:16:47.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-26T00:16:47.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:16:47.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-26T00:16:47.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:16:47.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:16:47.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:16:47.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:16:47.964 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:16:47.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:16:47.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:16:48.202 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:16:48.200+0000 7f2f11bec7c0 -1 Falling back to public interface 2024-03-26T00:16:48.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:16:49.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:16:49.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:16:49.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:16:49.278 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:16:49.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:16:49.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:16:49.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:16:50.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:16:50.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:16:50.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:16:50.596 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:16:50.597 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:16:50.597 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:16:50.769 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:16:50.768+0000 7f2f11bec7c0 -1 osd.0 101 log_to_monitors true 2024-03-26T00:16:50.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:16:51.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:16:51.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:16:51.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:16:51.917 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:16:51.918 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:16:51.918 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:16:52.267 INFO:tasks.workunit.client.0.smithi129.stderr:osd.0 up in weight 1 up_from 105 up_thru 105 down_at 102 last_clean_interval [93,101) [v2:127.0.0.1:6802/2215742840,v1:127.0.0.1:6803/2215742840] [v2:127.0.0.1:6804/2215742840,v1:127.0.0.1:6805/2215742840] exists,up d7a8ca59-f9ab-4120-b1c3-f87ea4b0bd36 2024-03-26T00:16:52.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:16:52.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:16:52.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:16:52.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:16:52.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:16:52.268 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:16:52.268 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:16:52.268 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:16:52.268 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:16:52.269 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:16:52.269 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:16:52.269 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:16:52.269 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:16:52.380 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:16:52.381 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:16:52.381 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:16:52.381 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:16:52.381 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:16:52.381 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:16:52.826 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:16:52.826 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:16:52.826 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:16:52.826 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:16:52.826 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:16:52.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566082 2024-03-26T00:16:52.974 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566082 2024-03-26T00:16:52.974 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566082' 2024-03-26T00:16:52.974 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:16:52.974 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:16:53.121 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=425201762309 2024-03-26T00:16:53.121 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 425201762309 2024-03-26T00:16:53.121 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566082 1-425201762309' 2024-03-26T00:16:53.121 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:16:53.122 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-450971566082 2024-03-26T00:16:53.122 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:16:53.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:16:53.124 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-450971566082 2024-03-26T00:16:53.124 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:16:53.125 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566082 2024-03-26T00:16:53.125 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 450971566082' 2024-03-26T00:16:53.125 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 450971566082 2024-03-26T00:16:53.125 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:16:53.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 450971566082 2024-03-26T00:16:53.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:16:54.411 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:16:54.412 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:16:54.724 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566082 -lt 450971566082 2024-03-26T00:16:54.724 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:16:54.725 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-425201762309 2024-03-26T00:16:54.725 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:16:54.726 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:16:54.727 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-425201762309 2024-03-26T00:16:54.727 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:16:54.728 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=425201762309 2024-03-26T00:16:54.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 425201762309' 2024-03-26T00:16:54.729 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 425201762309 2024-03-26T00:16:54.729 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:16:55.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 425201762310 -lt 425201762309 2024-03-26T00:16:55.091 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:16:55.091 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:16:55.091 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:16:55.456 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:16:55.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:16:55.457 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:16:55.458 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:16:55.458 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:16:55.458 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:16:55.458 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:16:55.458 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:16:55.745 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:16:55.745 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:16:55.746 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:16:55.746 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:16:56.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:16:56.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:16:56.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:16:56.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1086: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-03-26T00:16:56.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:16:56.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:16:56.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-26T00:16:56.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:16:56.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-03-26T00:16:56.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:16:56.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:16:56.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-26T00:16:56.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:16:56.167 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-26T00:16:56.167 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:16:56.167 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:16:56.167 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:16:56.167 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:16:56.167 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:16:56.475 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:16:56.476 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-03-26T00:16:56.476 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:16:56.476 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:16:56.476 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:16:56.476 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:16:56.476 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:16:56.476 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 rm-attr _key2-ROBJ8 2024-03-26T00:16:58.895 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-26T00:16:58.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:16:58.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:16:58.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-26T00:16:58.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:16:58.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:16:58.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:16:58.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:16:58.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:16:58.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:16:58.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:16:58.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:16:58.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:16:58.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:16:58.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:16:58.897 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:16:58.897 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:16:58.897 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:16:58.897 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:16:58.897 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:16:58.897 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:16:58.898 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:16:58.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:16:58.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:16:58.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:16:58.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:16:58.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:16:58.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:16:58.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:16:58.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:16:58.900 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:16:58.900 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:16:58.900 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:16:58.900 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-26T00:16:58.900 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.1 2024-03-26T00:16:58.900 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:16:58.901 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-26T00:16:58.901 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-26T00:16:58.901 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:16:58.901 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:16:58.902 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:16:58.935 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:16:58.932+0000 7f73d8f9d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:16:58.937 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:16:58.936+0000 7f73d8f9d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:16:58.939 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:16:58.940+0000 7f73d8f9d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:16:59.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-26T00:16:59.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:16:59.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:16:59.266 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:16:59.266 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:16:59.266 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:16:59.266 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:16:59.266 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:16:59.266 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:16:59.266 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:16:59.473 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:16:59.472+0000 7f73d8f9d7c0 -1 Falling back to public interface 2024-03-26T00:16:59.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:17:00.574 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:17:00.574 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:17:00.574 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:17:00.574 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:17:00.575 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:17:00.575 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:17:00.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:17:01.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:17:01.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:17:01.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:17:01.899 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:17:01.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:17:01.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:17:02.143 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:17:02.144+0000 7f73d8f9d7c0 -1 osd.1 106 log_to_monitors true 2024-03-26T00:17:02.211 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:17:03.213 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:17:03.213 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:17:03.213 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:17:03.213 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:17:03.214 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:17:03.214 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:17:03.620 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1 up in weight 1 up_from 110 up_thru 110 down_at 107 last_clean_interval [99,106) [v2:127.0.0.1:6810/3116086235,v1:127.0.0.1:6811/3116086235] [v2:127.0.0.1:6812/3116086235,v1:127.0.0.1:6813/3116086235] exists,up 0c2624cc-3bc6-4489-918a-a2155ff79e95 2024-03-26T00:17:03.620 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:17:03.620 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:17:03.620 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:17:03.620 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:17:03.620 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:17:03.620 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:17:03.620 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:17:03.621 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:17:03.621 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:17:03.621 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:17:03.622 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:17:03.622 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:17:03.622 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:17:03.741 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:17:03.741 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:17:03.741 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:17:03.741 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:17:03.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:17:03.742 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:17:04.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:17:04.157 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:17:04.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:17:04.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:17:04.158 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:17:04.322 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566085 2024-03-26T00:17:04.322 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566085 2024-03-26T00:17:04.322 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566085' 2024-03-26T00:17:04.322 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:17:04.323 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:17:04.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=472446402562 2024-03-26T00:17:04.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 472446402562 2024-03-26T00:17:04.573 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566085 1-472446402562' 2024-03-26T00:17:04.573 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:17:04.573 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-450971566085 2024-03-26T00:17:04.573 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:17:04.573 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:17:04.574 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-450971566085 2024-03-26T00:17:04.574 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:17:04.575 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566085 2024-03-26T00:17:04.575 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 450971566085' 2024-03-26T00:17:04.575 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 450971566085 2024-03-26T00:17:04.575 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:17:04.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566084 -lt 450971566085 2024-03-26T00:17:04.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:17:05.893 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:17:05.894 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:17:06.214 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566085 -lt 450971566085 2024-03-26T00:17:06.215 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:17:06.216 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-472446402562 2024-03-26T00:17:06.216 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:17:06.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:17:06.218 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-472446402562 2024-03-26T00:17:06.218 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:17:06.219 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=472446402562 2024-03-26T00:17:06.220 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 472446402562' 2024-03-26T00:17:06.220 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 472446402562 2024-03-26T00:17:06.220 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:17:06.516 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 472446402562 2024-03-26T00:17:06.516 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:17:07.519 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-26T00:17:07.519 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:17:07.818 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 472446402562 -lt 472446402562 2024-03-26T00:17:07.819 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:17:07.820 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:17:07.820 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:17:08.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:17:08.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:17:08.202 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:17:08.202 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:17:08.202 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:17:08.202 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:17:08.203 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:17:08.203 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:17:08.570 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:17:08.572 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:17:08.572 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:17:08.573 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:17:08.953 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:17:08.954 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:17:08.954 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:17:08.954 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1087: TEST_corrupt_scrub_replicated: echo -n val3-ROBJ8 2024-03-26T00:17:08.954 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-03-26T00:17:08.954 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:17:08.954 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:17:08.954 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-26T00:17:08.954 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:17:08.954 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-03-26T00:17:08.954 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:17:08.954 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:17:08.955 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-26T00:17:08.955 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:17:08.955 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-26T00:17:08.955 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:17:08.955 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:17:08.955 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:17:08.955 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:17:08.955 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:17:10.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:17:10.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-03-26T00:17:10.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:17:10.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:17:10.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:17:10.266 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:17:10.266 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:17:10.266 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-03-26T00:17:12.375 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-26T00:17:12.376 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:17:12.376 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:17:12.376 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-26T00:17:12.376 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:17:12.376 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:17:12.376 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:17:12.377 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:17:12.377 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:17:12.377 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:17:12.377 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:17:12.377 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:17:12.377 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:17:12.377 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:17:12.378 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:17:12.378 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:17:12.378 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:17:12.378 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:17:12.378 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:17:12.378 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:17:12.378 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:17:12.378 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:17:12.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:17:12.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:17:12.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:17:12.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:17:12.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:17:12.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:17:12.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:17:12.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:17:12.380 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:17:12.380 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:17:12.380 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:17:12.380 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-26T00:17:12.380 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.1 2024-03-26T00:17:12.380 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:17:12.380 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-26T00:17:12.380 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-26T00:17:12.381 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:17:12.381 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:17:12.381 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:17:12.407 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:17:12.404+0000 7f940f5817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:17:12.410 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:17:12.408+0000 7f940f5817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:17:12.411 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:17:12.412+0000 7f940f5817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:17:12.737 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-26T00:17:12.737 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:17:12.737 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:17:12.737 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:17:12.737 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:17:12.737 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:17:12.737 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:17:12.737 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:17:12.737 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:17:12.737 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:17:12.945 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:17:12.944+0000 7f940f5817c0 -1 Falling back to public interface 2024-03-26T00:17:13.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:17:14.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:17:14.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:17:14.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:17:14.048 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:17:14.049 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:17:14.049 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:17:14.366 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:17:15.368 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:17:15.368 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:17:15.368 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:17:15.368 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:17:15.369 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:17:15.369 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:17:15.674 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:17:15.672+0000 7f940f5817c0 -1 osd.1 112 log_to_monitors true 2024-03-26T00:17:15.695 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:17:16.696 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:17:16.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:17:16.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:17:16.697 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:17:16.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:17:16.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:17:17.205 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1 up in weight 1 up_from 116 up_thru 110 down_at 113 last_clean_interval [110,112) [v2:127.0.0.1:6810/2804702607,v1:127.0.0.1:6811/2804702607] [v2:127.0.0.1:6812/2804702607,v1:127.0.0.1:6813/2804702607] exists,up 0c2624cc-3bc6-4489-918a-a2155ff79e95 2024-03-26T00:17:17.205 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:17:17.205 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:17:17.205 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:17:17.206 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:17:17.206 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:17:17.206 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:17:17.206 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:17:17.207 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:17:17.207 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:17:17.207 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:17:17.207 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:17:17.207 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:17:17.207 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:17:17.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:17:17.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:17:17.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:17:17.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:17:17.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:17:17.330 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:17:17.669 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:17:17.669 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:17:17.669 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:17:17.669 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:17:17.669 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:17:17.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566089 2024-03-26T00:17:17.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566089 2024-03-26T00:17:17.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566089' 2024-03-26T00:17:17.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:17:17.836 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:17:18.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=498216206338 2024-03-26T00:17:18.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 498216206338 2024-03-26T00:17:18.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566089 1-498216206338' 2024-03-26T00:17:18.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:17:18.034 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-450971566089 2024-03-26T00:17:18.034 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:17:18.035 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:17:18.036 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-450971566089 2024-03-26T00:17:18.036 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:17:18.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566089 2024-03-26T00:17:18.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 450971566089' 2024-03-26T00:17:18.038 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 450971566089 2024-03-26T00:17:18.038 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:17:18.340 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566088 -lt 450971566089 2024-03-26T00:17:18.340 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:17:19.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:17:19.342 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:17:19.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566089 -lt 450971566089 2024-03-26T00:17:19.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:17:19.655 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-498216206338 2024-03-26T00:17:19.655 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:17:19.656 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:17:19.657 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-498216206338 2024-03-26T00:17:19.657 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:17:19.658 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=498216206338 2024-03-26T00:17:19.658 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 498216206338' 2024-03-26T00:17:19.659 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 498216206338 2024-03-26T00:17:19.659 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:17:19.976 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 498216206338 -lt 498216206338 2024-03-26T00:17:19.977 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:17:19.977 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:17:19.977 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:17:20.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:17:20.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:17:20.333 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:17:20.333 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:17:20.333 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:17:20.333 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:17:20.333 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:17:20.334 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:17:20.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-26T00:17:20.631 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:17:20.632 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:17:20.632 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:17:20.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-26T00:17:20.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-26T00:17:20.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-26T00:17:20.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-26T00:17:20.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-26T00:17:20.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-26T00:17:21.080 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-26T00:17:21.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:17:21.081 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:17:21.081 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:17:21.081 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:17:21.081 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:17:21.082 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:17:21.082 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:17:21.372 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-26T00:17:21.372 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:17:21.373 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:17:21.373 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:17:21.721 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-26T00:17:21.722 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-26T00:17:21.722 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-26T00:17:21.722 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-26T00:17:21.722 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-26T00:17:21.722 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-26T00:17:21.722 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-26T00:17:21.722 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-26T00:17:21.722 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-26T00:17:22.084 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-26T00:17:22.084 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-26T00:17:22.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-26T00:17:22.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-26T00:17:22.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-26T00:17:22.285 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-26T00:17:22.285 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:17:22.286 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:17:22.286 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:17:22.286 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:17:22.286 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:17:22.287 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:17:22.287 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:17:22.583 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:17:22.584 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:17:22.584 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:17:22.585 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:17:22.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:17:22.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:17:22.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:17:22.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-03-26T00:17:22.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-26T00:17:22.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-03-26T00:17:22.965 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 9 % 2 2024-03-26T00:17:22.968 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-26T00:17:22.968 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-26T00:17:22.968 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1093: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-03-26T00:17:22.968 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:17:22.968 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:17:22.968 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-26T00:17:22.968 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:17:22.968 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-03-26T00:17:22.968 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:17:22.968 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:17:22.968 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-26T00:17:22.968 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:17:22.969 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-26T00:17:22.969 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:17:22.969 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:17:22.969 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:17:22.969 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:17:22.969 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:17:24.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:17:24.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-03-26T00:17:24.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:17:24.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:17:24.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:17:24.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:17:24.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:17:24.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 get-attr _ 2024-03-26T00:17:26.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-26T00:17:26.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:17:26.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:17:26.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-26T00:17:26.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:17:26.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:17:26.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:17:26.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:17:26.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:17:26.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:17:26.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:17:26.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:17:26.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:17:26.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:17:26.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:17:26.388 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:17:26.388 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:17:26.388 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:17:26.390 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:17:26.390 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:17:26.391 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:17:26.391 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:17:26.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:17:26.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:17:26.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:17:26.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:17:26.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:17:26.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:17:26.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:17:26.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:17:26.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:17:26.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:17:26.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:17:26.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-26T00:17:26.392 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.1 2024-03-26T00:17:26.393 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:17:26.393 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-26T00:17:26.393 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-26T00:17:26.393 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:17:26.393 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:17:26.393 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:17:26.438 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:17:26.435+0000 7fa4b44f17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:17:26.441 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:17:26.439+0000 7fa4b44f17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:17:26.442 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:17:26.439+0000 7fa4b44f17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:17:26.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-26T00:17:26.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:17:26.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:17:26.717 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:17:26.717 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:17:26.717 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:17:26.717 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:17:26.717 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:17:26.717 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:17:26.717 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:17:26.986 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:17:26.983+0000 7fa4b44f17c0 -1 Falling back to public interface 2024-03-26T00:17:27.030 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:17:28.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:17:28.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:17:28.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:17:28.032 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:17:28.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:17:28.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:17:28.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:17:29.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:17:29.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:17:29.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:17:29.344 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:17:29.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:17:29.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:17:29.644 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:17:29.643+0000 7fa4b44f17c0 -1 osd.1 117 log_to_monitors true 2024-03-26T00:17:29.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:17:30.668 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:17:30.669 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:17:30.669 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:17:30.669 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:17:30.669 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:17:30.669 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:17:31.036 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1 up in weight 1 up_from 121 up_thru 116 down_at 118 last_clean_interval [116,117) [v2:127.0.0.1:6810/262478295,v1:127.0.0.1:6811/262478295] [v2:127.0.0.1:6812/262478295,v1:127.0.0.1:6813/262478295] exists,up 0c2624cc-3bc6-4489-918a-a2155ff79e95 2024-03-26T00:17:31.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:17:31.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:17:31.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:17:31.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:17:31.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:17:31.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:17:31.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:17:31.037 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:17:31.037 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:17:31.037 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:17:31.038 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:17:31.038 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:17:31.038 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:17:31.156 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:17:31.156 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:17:31.156 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:17:31.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:17:31.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:17:31.157 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:17:31.455 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:17:31.455 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:17:31.455 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:17:31.455 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:17:31.455 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:17:31.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566093 2024-03-26T00:17:31.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566093 2024-03-26T00:17:31.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566093' 2024-03-26T00:17:31.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:17:31.659 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:17:31.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=519691042818 2024-03-26T00:17:31.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 519691042818 2024-03-26T00:17:31.837 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566093 1-519691042818' 2024-03-26T00:17:31.837 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:17:31.837 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-450971566093 2024-03-26T00:17:31.838 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:17:31.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:17:31.840 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-450971566093 2024-03-26T00:17:31.840 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:17:31.841 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566093 2024-03-26T00:17:31.841 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 450971566093' 2024-03-26T00:17:31.841 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 450971566093 2024-03-26T00:17:31.841 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:17:32.146 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566091 -lt 450971566093 2024-03-26T00:17:32.147 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:17:33.148 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:17:33.149 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:17:33.451 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566093 -lt 450971566093 2024-03-26T00:17:33.451 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:17:33.452 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-519691042818 2024-03-26T00:17:33.453 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:17:33.455 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:17:33.456 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-519691042818 2024-03-26T00:17:33.456 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:17:33.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=519691042818 2024-03-26T00:17:33.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 519691042818' 2024-03-26T00:17:33.457 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 519691042818 2024-03-26T00:17:33.458 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:17:33.765 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 519691042818 -lt 519691042818 2024-03-26T00:17:33.766 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:17:33.767 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:17:33.767 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:17:34.120 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:17:34.120 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:17:34.121 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:17:34.121 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:17:34.122 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:17:34.122 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:17:34.122 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:17:34.122 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:17:34.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-26T00:17:34.421 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:17:34.421 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:17:34.421 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:17:34.775 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-26T00:17:34.775 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-26T00:17:34.775 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-26T00:17:34.775 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-26T00:17:34.775 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-26T00:17:34.775 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-26T00:17:34.876 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-26T00:17:34.876 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:17:34.877 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:17:34.877 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:17:34.877 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:17:34.877 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:17:34.878 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:17:34.878 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:17:35.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-26T00:17:35.232 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:17:35.233 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:17:35.233 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:17:35.590 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-26T00:17:35.590 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-26T00:17:35.590 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-26T00:17:35.590 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-26T00:17:35.590 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-26T00:17:35.590 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-26T00:17:35.590 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-26T00:17:35.591 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-26T00:17:35.591 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-26T00:17:35.970 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-26T00:17:35.970 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-26T00:17:35.970 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-26T00:17:35.970 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-26T00:17:35.970 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-26T00:17:36.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-26T00:17:36.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:17:36.174 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:17:36.174 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:17:36.174 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:17:36.174 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:17:36.175 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:17:36.175 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:17:36.473 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-26T00:17:36.474 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:17:36.474 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:17:36.474 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:17:36.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-26T00:17:36.864 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-26T00:17:36.864 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-26T00:17:36.864 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-26T00:17:36.864 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-26T00:17:36.864 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-26T00:17:36.864 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-26T00:17:36.864 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-26T00:17:36.864 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-26T00:17:37.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-26T00:17:37.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-26T00:17:37.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-26T00:17:37.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-26T00:17:37.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-26T00:17:37.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-26T00:17:37.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:17:37.631 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:17:37.631 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:17:37.631 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:17:37.631 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:17:37.632 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:17:37.632 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:17:37.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:17:37.927 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:17:37.927 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:17:37.928 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:17:38.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:17:38.351 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:17:38.351 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:17:38.351 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: echo -n D 2024-03-26T00:17:38.351 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1095: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2024-03-26T00:17:38.545 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1096: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-03-26T00:17:38.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:17:38.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:17:38.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-26T00:17:38.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:17:38.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-03-26T00:17:38.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:17:38.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:17:38.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-26T00:17:38.547 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:17:38.547 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-26T00:17:38.547 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:17:38.547 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:17:38.548 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:17:38.548 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:17:38.548 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:17:39.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:17:39.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-03-26T00:17:39.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:17:39.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:17:39.864 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:17:39.864 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:17:39.864 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:17:39.864 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-03-26T00:17:41.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-26T00:17:41.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:17:41.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:17:41.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-26T00:17:41.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:17:41.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:17:41.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:17:41.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:17:41.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:17:41.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:17:41.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:17:41.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:17:41.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:17:41.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:17:41.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:17:41.781 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:17:41.781 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:17:41.781 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:17:41.781 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:17:41.781 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:17:41.781 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:17:41.782 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:17:41.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:17:41.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:17:41.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:17:41.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:17:41.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:17:41.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:17:41.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:17:41.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:17:41.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:17:41.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:17:41.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:17:41.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-26T00:17:41.783 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.1 2024-03-26T00:17:41.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:17:41.784 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-26T00:17:41.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-26T00:17:41.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:17:41.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:17:41.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:17:41.814 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:17:41.811+0000 7fd0b2de17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:17:41.817 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:17:41.815+0000 7fd0b2de17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:17:41.818 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:17:41.815+0000 7fd0b2de17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:17:42.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-26T00:17:42.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:17:42.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:17:42.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:17:42.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:17:42.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:17:42.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:17:42.097 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:17:42.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:17:42.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:17:42.362 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:17:42.359+0000 7fd0b2de17c0 -1 Falling back to public interface 2024-03-26T00:17:42.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:17:43.411 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:17:43.411 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:17:43.411 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:17:43.411 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:17:43.412 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:17:43.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:17:43.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:17:44.733 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:17:44.733 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:17:44.733 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:17:44.733 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:17:44.733 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:17:44.733 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:17:45.013 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:17:45.011+0000 7fd0b2de17c0 -1 osd.1 124 log_to_monitors true 2024-03-26T00:17:45.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:17:46.096 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:17:46.096 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:17:46.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:17:46.097 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:17:46.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:17:46.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:17:46.426 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1 up in weight 1 up_from 127 up_thru 127 down_at 124 last_clean_interval [121,124) [v2:127.0.0.1:6810/860723047,v1:127.0.0.1:6811/860723047] [v2:127.0.0.1:6812/860723047,v1:127.0.0.1:6813/860723047] exists,up 0c2624cc-3bc6-4489-918a-a2155ff79e95 2024-03-26T00:17:46.426 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:17:46.426 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:17:46.426 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:17:46.426 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:17:46.426 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:17:46.426 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:17:46.426 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:17:46.427 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:17:46.427 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:17:46.427 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:17:46.428 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:17:46.428 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:17:46.428 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:17:46.545 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:17:46.545 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:17:46.545 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:17:46.545 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:17:46.545 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:17:46.546 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:17:46.891 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:17:46.892 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:17:46.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:17:46.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:17:46.892 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:17:47.053 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566097 2024-03-26T00:17:47.053 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566097 2024-03-26T00:17:47.053 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566097' 2024-03-26T00:17:47.053 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:17:47.053 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:17:47.200 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=545460846594 2024-03-26T00:17:47.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 545460846594 2024-03-26T00:17:47.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566097 1-545460846594' 2024-03-26T00:17:47.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:17:47.201 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-450971566097 2024-03-26T00:17:47.201 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:17:47.203 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:17:47.204 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-450971566097 2024-03-26T00:17:47.204 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:17:47.205 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566097 2024-03-26T00:17:47.205 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 450971566097' 2024-03-26T00:17:47.205 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 450971566097 2024-03-26T00:17:47.205 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:17:47.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566096 -lt 450971566097 2024-03-26T00:17:47.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:17:48.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:17:48.513 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:17:48.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566097 -lt 450971566097 2024-03-26T00:17:48.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:17:48.837 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-545460846594 2024-03-26T00:17:48.838 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:17:48.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:17:48.841 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-545460846594 2024-03-26T00:17:48.841 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:17:48.842 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=545460846594 2024-03-26T00:17:48.843 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 545460846594' 2024-03-26T00:17:48.844 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 545460846594 2024-03-26T00:17:48.844 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:17:49.180 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 545460846594 -lt 545460846594 2024-03-26T00:17:49.181 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:17:49.181 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:17:49.181 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:17:49.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:17:49.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:17:49.563 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:17:49.563 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:17:49.563 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:17:49.563 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:17:49.563 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:17:49.563 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:17:49.889 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:17:49.890 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:17:49.890 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:17:49.890 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:17:50.252 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:17:50.252 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:17:50.252 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:17:50.252 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1097: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2024-03-26T00:17:50.252 INFO:tasks.workunit.client.0.smithi129.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2024-03-26T00:17:50.252 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-26T00:17:50.253 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-03-26T00:17:50.253 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 10 % 2 2024-03-26T00:17:50.253 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-26T00:17:50.253 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-26T00:17:50.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-26T00:17:50.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ11 2024-03-26T00:17:50.254 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 11 % 2 2024-03-26T00:17:50.255 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-26T00:17:50.255 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-26T00:17:50.255 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-26T00:17:50.255 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ12 2024-03-26T00:17:50.255 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 12 % 2 2024-03-26T00:17:50.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-26T00:17:50.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-26T00:17:50.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-26T00:17:50.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ13 2024-03-26T00:17:50.257 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 13 % 2 2024-03-26T00:17:50.259 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-26T00:17:50.259 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-26T00:17:50.259 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-26T00:17:50.259 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ14 2024-03-26T00:17:50.259 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 14 % 2 2024-03-26T00:17:50.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-26T00:17:50.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-26T00:17:50.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1106: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-03-26T00:17:50.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1107: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-03-26T00:17:50.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:17:50.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:17:50.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-26T00:17:50.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:17:50.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-03-26T00:17:50.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:17:50.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:17:50.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-26T00:17:50.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:17:50.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-26T00:17:50.263 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:17:50.263 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:17:50.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:17:50.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:17:50.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:17:50.571 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:17:50.571 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-03-26T00:17:50.571 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:17:50.571 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:17:50.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:17:50.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:17:50.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:17:50.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-03-26T00:17:52.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-26T00:17:52.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:17:52.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:17:52.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-26T00:17:52.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:17:52.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:17:52.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:17:52.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:17:52.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:17:52.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:17:52.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-26T00:17:52.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-26T00:17:52.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:17:52.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:17:52.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:17:52.499 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:17:52.499 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:17:52.499 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:17:52.501 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:17:52.501 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:17:52.501 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:17:52.501 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:17:52.501 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:17:52.501 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:17:52.501 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:17:52.501 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:17:52.501 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:17:52.501 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:17:52.501 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:17:52.501 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:17:52.501 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:17:52.501 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:17:52.501 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-26T00:17:52.502 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-26T00:17:52.502 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.0 2024-03-26T00:17:52.502 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:17:52.502 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-26T00:17:52.502 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-26T00:17:52.503 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:17:52.503 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:17:52.503 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:17:52.535 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:17:52.531+0000 7f16670ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:17:52.537 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:17:52.535+0000 7f16670ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:17:52.539 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:17:52.535+0000 7f16670ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:17:52.828 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-26T00:17:52.828 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:17:52.828 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-26T00:17:52.828 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:17:52.828 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:17:52.828 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:17:52.828 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:17:52.829 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:17:52.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:17:52.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:17:53.081 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:17:53.079+0000 7f16670ec7c0 -1 Falling back to public interface 2024-03-26T00:17:53.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:17:54.187 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:17:54.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:17:54.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:17:54.188 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:17:54.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:17:54.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:17:54.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:17:55.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:17:55.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:17:55.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:17:55.499 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:17:55.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:17:55.500 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:17:55.685 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:17:55.683+0000 7f16670ec7c0 -1 osd.0 129 log_to_monitors true 2024-03-26T00:17:55.825 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:17:56.827 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:17:56.828 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:17:56.828 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:17:56.828 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:17:56.828 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:17:56.828 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:17:57.253 INFO:tasks.workunit.client.0.smithi129.stderr:osd.0 up in weight 1 up_from 133 up_thru 133 down_at 130 last_clean_interval [105,129) [v2:127.0.0.1:6802/3195600701,v1:127.0.0.1:6803/3195600701] [v2:127.0.0.1:6804/3195600701,v1:127.0.0.1:6805/3195600701] exists,up d7a8ca59-f9ab-4120-b1c3-f87ea4b0bd36 2024-03-26T00:17:57.253 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:17:57.253 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:17:57.253 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:17:57.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:17:57.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:17:57.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:17:57.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:17:57.254 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:17:57.256 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:17:57.256 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:17:57.256 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:17:57.256 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:17:57.257 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:17:57.377 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:17:57.377 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:17:57.377 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:17:57.377 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:17:57.377 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:17:57.377 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:17:57.709 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:17:57.709 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:17:57.709 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:17:57.709 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:17:57.710 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:17:57.886 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=571230650370 2024-03-26T00:17:57.886 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 571230650370 2024-03-26T00:17:57.886 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-571230650370' 2024-03-26T00:17:57.886 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:17:57.886 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:17:58.030 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=545460846597 2024-03-26T00:17:58.030 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 545460846597 2024-03-26T00:17:58.030 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-571230650370 1-545460846597' 2024-03-26T00:17:58.030 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:17:58.030 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:17:58.031 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-571230650370 2024-03-26T00:17:58.031 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:17:58.032 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-571230650370 2024-03-26T00:17:58.032 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:17:58.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=571230650370 2024-03-26T00:17:58.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 571230650370' 2024-03-26T00:17:58.034 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 571230650370 2024-03-26T00:17:58.034 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:17:58.327 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 571230650370 2024-03-26T00:17:58.328 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:17:59.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:17:59.330 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:17:59.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 571230650370 -lt 571230650370 2024-03-26T00:17:59.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:17:59.653 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-545460846597 2024-03-26T00:17:59.653 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:17:59.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:17:59.654 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-545460846597 2024-03-26T00:17:59.654 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:17:59.655 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=545460846597 2024-03-26T00:17:59.655 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 545460846597' 2024-03-26T00:17:59.655 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 545460846597 2024-03-26T00:17:59.656 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:17:59.994 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 545460846597 -lt 545460846597 2024-03-26T00:17:59.995 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:17:59.996 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:17:59.996 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:18:00.384 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:18:00.384 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:18:00.385 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:18:00.385 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:18:00.385 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:18:00.385 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:18:00.386 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:18:00.386 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:18:00.759 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:18:00.759 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:18:00.759 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:18:00.760 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:18:01.148 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:18:01.149 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:18:01.149 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:18:01.149 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1108: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-03-26T00:18:01.149 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:18:01.149 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:18:01.149 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-26T00:18:01.149 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:18:01.149 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-03-26T00:18:01.149 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:18:01.149 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:18:01.149 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-26T00:18:01.149 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:18:01.150 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-26T00:18:01.150 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:18:01.150 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:18:01.150 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:18:01.150 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:18:01.150 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:18:01.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:18:01.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-03-26T00:18:01.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:18:01.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:18:01.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:18:01.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:18:01.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:18:01.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ14 rm-attr _ 2024-03-26T00:18:03.191 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-26T00:18:03.191 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:18:03.191 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:18:03.191 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-26T00:18:03.191 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:18:03.191 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:18:03.191 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:18:03.191 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:18:03.192 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:18:03.192 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:18:03.192 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:18:03.192 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:18:03.192 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:18:03.192 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:18:03.192 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:18:03.192 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:18:03.192 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:18:03.192 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:18:03.193 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:18:03.194 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:18:03.194 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:18:03.194 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:18:03.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:18:03.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:18:03.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:18:03.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:18:03.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:18:03.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:18:03.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:18:03.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:18:03.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:18:03.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:18:03.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:18:03.195 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-26T00:18:03.195 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.1 2024-03-26T00:18:03.195 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:18:03.195 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-26T00:18:03.196 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-26T00:18:03.196 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:18:03.196 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:18:03.196 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:18:03.229 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:18:03.227+0000 7fcc786c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:18:03.231 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:18:03.231+0000 7fcc786c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:18:03.233 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:18:03.231+0000 7fcc786c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:18:03.518 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-26T00:18:03.518 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:18:03.518 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:18:03.518 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:18:03.518 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:18:03.518 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:18:03.518 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:18:03.519 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:18:03.519 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:18:03.519 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:18:03.766 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:18:03.763+0000 7fcc786c37c0 -1 Falling back to public interface 2024-03-26T00:18:03.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:18:04.871 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:18:04.871 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:18:04.871 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:18:04.871 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:18:04.871 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:18:04.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:18:05.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:18:06.190 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:18:06.190 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:18:06.190 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:18:06.191 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:18:06.191 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:18:06.191 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:18:06.430 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:18:06.426+0000 7fcc786c37c0 -1 osd.1 134 log_to_monitors true 2024-03-26T00:18:06.551 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:18:07.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:18:07.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:18:07.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:18:07.553 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:18:07.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:18:07.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:18:08.020 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1 up in weight 1 up_from 138 up_thru 138 down_at 135 last_clean_interval [127,134) [v2:127.0.0.1:6810/2812234352,v1:127.0.0.1:6811/2812234352] [v2:127.0.0.1:6812/2812234352,v1:127.0.0.1:6813/2812234352] exists,up 0c2624cc-3bc6-4489-918a-a2155ff79e95 2024-03-26T00:18:08.020 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:18:08.020 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:18:08.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:18:08.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:18:08.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:18:08.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:18:08.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:18:08.022 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:18:08.022 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:18:08.022 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:18:08.022 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:18:08.022 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:18:08.022 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:18:08.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:18:08.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:18:08.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:18:08.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:18:08.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:18:08.132 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:18:08.445 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:18:08.446 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:18:08.446 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:18:08.446 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:18:08.446 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:18:08.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=571230650373 2024-03-26T00:18:08.696 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 571230650373 2024-03-26T00:18:08.696 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-571230650373' 2024-03-26T00:18:08.696 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:18:08.696 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:18:08.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=592705486850 2024-03-26T00:18:08.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 592705486850 2024-03-26T00:18:08.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-571230650373 1-592705486850' 2024-03-26T00:18:08.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:18:08.821 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-571230650373 2024-03-26T00:18:08.821 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:18:08.821 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:18:08.822 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-571230650373 2024-03-26T00:18:08.822 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:18:08.824 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=571230650373 2024-03-26T00:18:08.824 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 571230650373' 2024-03-26T00:18:08.824 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 571230650373 2024-03-26T00:18:08.824 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:18:09.151 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 571230650372 -lt 571230650373 2024-03-26T00:18:09.151 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:18:10.152 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:18:10.153 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:18:10.478 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 571230650372 -lt 571230650373 2024-03-26T00:18:10.478 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:18:11.483 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-26T00:18:11.483 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:18:11.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 571230650373 -lt 571230650373 2024-03-26T00:18:11.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:18:11.801 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:18:11.801 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-592705486850 2024-03-26T00:18:11.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:18:11.803 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-592705486850 2024-03-26T00:18:11.803 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:18:11.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=592705486850 2024-03-26T00:18:11.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 592705486850' 2024-03-26T00:18:11.805 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 592705486850 2024-03-26T00:18:11.805 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:18:12.127 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 592705486850 -lt 592705486850 2024-03-26T00:18:12.127 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:18:12.128 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:18:12.128 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:18:12.533 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:18:12.533 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:18:12.533 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:18:12.533 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:18:12.533 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:18:12.533 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:18:12.533 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:18:12.533 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:18:12.879 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:18:12.879 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:18:12.879 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:18:12.879 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:18:13.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:18:13.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:18:13.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:18:13.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1109: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val 2024-03-26T00:18:13.248 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-26T00:18:13.248 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ15 2024-03-26T00:18:13.248 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 15 % 2 2024-03-26T00:18:13.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-26T00:18:13.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-26T00:18:13.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1113: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-03-26T00:18:13.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:18:13.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:18:13.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-26T00:18:13.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:18:13.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-03-26T00:18:13.255 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:18:13.255 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:18:13.255 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-26T00:18:13.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:18:13.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-26T00:18:13.256 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:18:13.256 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:18:13.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:18:13.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:18:13.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:18:13.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:18:13.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-03-26T00:18:13.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:18:13.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:18:13.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:18:13.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:18:13.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:18:13.561 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ15 rm-attr _ 2024-03-26T00:18:14.862 INFO:tasks.workunit.client.0.smithi129.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-03-26T00:18:15.399 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-26T00:18:15.399 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:18:15.399 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:18:15.399 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-26T00:18:15.399 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:18:15.400 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:18:15.400 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:18:15.400 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:18:15.400 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:18:15.400 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:18:15.400 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:18:15.400 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:18:15.400 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:18:15.400 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:18:15.400 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:18:15.401 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:18:15.401 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:18:15.401 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:18:15.403 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:18:15.403 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:18:15.403 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:18:15.404 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:18:15.404 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:18:15.404 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:18:15.404 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:18:15.404 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:18:15.404 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:18:15.404 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:18:15.404 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:18:15.404 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:18:15.405 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:18:15.405 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:18:15.405 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:18:15.405 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-26T00:18:15.405 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.1 2024-03-26T00:18:15.405 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:18:15.406 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-26T00:18:15.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-26T00:18:15.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:18:15.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:18:15.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:18:15.442 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:18:15.438+0000 7f3aafeeb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:18:15.445 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:18:15.442+0000 7f3aafeeb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:18:15.447 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:18:15.442+0000 7f3aafeeb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:18:15.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-26T00:18:15.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:18:15.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:18:15.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:18:15.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:18:15.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:18:15.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:18:15.713 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:18:15.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:18:15.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:18:15.990 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:18:15.986+0000 7f3aafeeb7c0 -1 Falling back to public interface 2024-03-26T00:18:16.041 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:18:17.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:18:17.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:18:17.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:18:17.043 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:18:17.044 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:18:17.044 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:18:17.376 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:18:18.377 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:18:18.377 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:18:18.377 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:18:18.377 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:18:18.378 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:18:18.378 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:18:18.486 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:18:18.482+0000 7f3aafeeb7c0 -1 osd.1 140 log_to_monitors true 2024-03-26T00:18:18.720 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:18:19.722 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:18:19.722 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:18:19.723 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:18:19.723 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:18:19.723 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:18:19.723 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:18:20.071 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1 up in weight 1 up_from 144 up_thru 144 down_at 141 last_clean_interval [138,140) [v2:127.0.0.1:6810/3157258800,v1:127.0.0.1:6811/3157258800] [v2:127.0.0.1:6812/3157258800,v1:127.0.0.1:6813/3157258800] exists,up 0c2624cc-3bc6-4489-918a-a2155ff79e95 2024-03-26T00:18:20.071 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:18:20.071 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:18:20.071 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:18:20.071 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:18:20.071 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:18:20.072 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:18:20.072 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:18:20.072 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:18:20.072 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:18:20.072 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:18:20.073 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:18:20.073 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:18:20.073 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:18:20.193 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:18:20.193 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:18:20.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:18:20.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:18:20.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:18:20.194 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:18:20.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:18:20.509 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:18:20.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:18:20.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:18:20.509 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:18:20.722 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=571230650376 2024-03-26T00:18:20.722 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 571230650376 2024-03-26T00:18:20.722 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-571230650376' 2024-03-26T00:18:20.723 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:18:20.723 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:18:20.865 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=618475290626 2024-03-26T00:18:20.865 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 618475290626 2024-03-26T00:18:20.865 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-571230650376 1-618475290626' 2024-03-26T00:18:20.865 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:18:20.866 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-571230650376 2024-03-26T00:18:20.866 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:18:20.867 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:18:20.868 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-571230650376 2024-03-26T00:18:20.869 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:18:20.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=571230650376 2024-03-26T00:18:20.869 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 571230650376' 2024-03-26T00:18:20.870 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 571230650376 2024-03-26T00:18:20.870 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:18:21.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 571230650375 -lt 571230650376 2024-03-26T00:18:21.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:18:22.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:18:22.179 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:18:22.501 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 571230650375 -lt 571230650376 2024-03-26T00:18:22.501 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:18:23.503 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-26T00:18:23.504 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:18:23.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 571230650377 -lt 571230650376 2024-03-26T00:18:23.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:18:23.830 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-618475290626 2024-03-26T00:18:23.830 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:18:23.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:18:23.834 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-618475290626 2024-03-26T00:18:23.834 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:18:23.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=618475290626 2024-03-26T00:18:23.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 618475290626' 2024-03-26T00:18:23.835 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 618475290626 2024-03-26T00:18:23.836 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:18:24.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 618475290626 -lt 618475290626 2024-03-26T00:18:24.176 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:18:24.177 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:18:24.177 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:18:24.555 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:18:24.555 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:18:24.556 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:18:24.556 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:18:24.556 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:18:24.556 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:18:24.556 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:18:24.556 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:18:24.870 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:18:24.967 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:18:24.967 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:18:24.967 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:18:25.224 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:18:25.224 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:18:25.224 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:18:25.224 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-26T00:18:25.224 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ16 2024-03-26T00:18:25.224 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 16 % 2 2024-03-26T00:18:25.226 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-26T00:18:25.226 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-26T00:18:25.226 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1117: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-03-26T00:18:25.226 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:18:25.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:18:25.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-26T00:18:25.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:18:25.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-03-26T00:18:25.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:18:25.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:18:25.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-26T00:18:25.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:18:25.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-26T00:18:25.227 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:18:25.227 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:18:25.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:18:25.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:18:25.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:18:25.536 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:18:25.536 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-03-26T00:18:25.536 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:18:25.536 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:18:25.536 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:18:25.536 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:18:25.536 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:18:25.536 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ16 rm-attr snapset 2024-03-26T00:18:26.826 INFO:tasks.workunit.client.0.smithi129.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-03-26T00:18:27.620 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-26T00:18:27.620 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:18:27.620 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:18:27.620 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-26T00:18:27.620 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:18:27.620 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:18:27.620 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:18:27.620 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:18:27.620 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:18:27.620 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:18:27.620 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-26T00:18:27.621 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-26T00:18:27.621 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:18:27.621 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:18:27.621 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:18:27.621 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:18:27.621 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:18:27.621 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:18:27.622 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:18:27.623 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:18:27.623 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:18:27.623 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:18:27.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:18:27.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:18:27.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:18:27.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:18:27.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:18:27.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:18:27.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:18:27.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:18:27.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:18:27.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:18:27.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-26T00:18:27.624 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-26T00:18:27.624 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.0 2024-03-26T00:18:27.624 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:18:27.624 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-26T00:18:27.625 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-26T00:18:27.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:18:27.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:18:27.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:18:27.658 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:18:27.654+0000 7f46193d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:18:27.661 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:18:27.658+0000 7f46193d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:18:27.662 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:18:27.658+0000 7f46193d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:18:27.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-26T00:18:27.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:18:27.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-26T00:18:27.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:18:27.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:18:27.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:18:27.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:18:27.949 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:18:27.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:18:27.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:18:28.213 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:18:28.206+0000 7f46193d37c0 -1 Falling back to public interface 2024-03-26T00:18:28.268 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:18:29.269 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:18:29.270 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:18:29.270 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:18:29.270 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:18:29.271 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:18:29.271 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:18:29.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:18:30.574 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:18:30.574 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:18:30.574 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:18:30.574 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:18:30.574 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:18:30.575 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:18:30.914 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:18:30.910+0000 7f46193d37c0 -1 osd.0 145 log_to_monitors true 2024-03-26T00:18:30.935 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:18:31.937 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:18:31.937 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:18:31.937 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:18:31.937 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:18:31.938 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:18:31.938 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:18:32.261 INFO:tasks.workunit.client.0.smithi129.stderr:osd.0 up in weight 1 up_from 149 up_thru 149 down_at 146 last_clean_interval [133,145) [v2:127.0.0.1:6802/1804748341,v1:127.0.0.1:6803/1804748341] [v2:127.0.0.1:6804/1804748341,v1:127.0.0.1:6805/1804748341] exists,up d7a8ca59-f9ab-4120-b1c3-f87ea4b0bd36 2024-03-26T00:18:32.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:18:32.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:18:32.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:18:32.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:18:32.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:18:32.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:18:32.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:18:32.262 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:18:32.263 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:18:32.263 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:18:32.263 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:18:32.263 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:18:32.263 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:18:32.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:18:32.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:18:32.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:18:32.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:18:32.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:18:32.382 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:18:32.694 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:18:32.695 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:18:32.695 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:18:32.695 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:18:32.695 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:18:32.845 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=639950127106 2024-03-26T00:18:32.845 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 639950127106 2024-03-26T00:18:32.845 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-639950127106' 2024-03-26T00:18:32.845 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:18:32.846 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:18:32.989 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=618475290629 2024-03-26T00:18:32.990 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 618475290629 2024-03-26T00:18:32.990 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-639950127106 1-618475290629' 2024-03-26T00:18:32.990 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:18:32.991 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-639950127106 2024-03-26T00:18:32.991 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:18:32.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:18:32.992 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-639950127106 2024-03-26T00:18:32.992 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:18:32.993 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=639950127106 2024-03-26T00:18:32.994 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 639950127106' 2024-03-26T00:18:32.994 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 639950127106 2024-03-26T00:18:32.994 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:18:33.324 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 639950127106 2024-03-26T00:18:33.324 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:18:34.325 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:18:34.326 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:18:34.658 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 639950127106 2024-03-26T00:18:34.658 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:18:35.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-26T00:18:35.661 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:18:35.968 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 639950127106 -lt 639950127106 2024-03-26T00:18:35.968 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:18:35.969 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:18:35.969 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-618475290629 2024-03-26T00:18:35.971 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:18:35.972 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-618475290629 2024-03-26T00:18:35.972 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:18:35.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=618475290629 2024-03-26T00:18:35.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 618475290629' 2024-03-26T00:18:35.973 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 618475290629 2024-03-26T00:18:35.973 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:18:36.286 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 618475290630 -lt 618475290629 2024-03-26T00:18:36.286 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:18:36.287 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:18:36.287 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:18:36.674 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:18:36.674 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:18:36.675 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:18:36.675 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:18:36.675 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:18:36.676 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:18:36.676 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:18:36.676 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:18:36.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:18:36.964 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:18:36.965 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:18:36.965 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:18:37.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:18:37.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:18:37.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:18:37.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1118: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-03-26T00:18:37.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1119: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-03-26T00:18:37.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:18:37.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:18:37.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-26T00:18:37.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:18:37.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-03-26T00:18:37.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:18:37.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:18:37.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-26T00:18:37.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:18:37.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-26T00:18:37.342 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:18:37.342 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:18:37.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:18:37.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:18:37.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:18:37.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:18:37.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-03-26T00:18:37.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:18:37.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:18:37.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:18:37.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:18:37.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:18:37.652 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-03-26T00:18:39.297 INFO:tasks.workunit.client.0.smithi129.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-03-26T00:18:39.298 INFO:tasks.workunit.client.0.smithi129.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-03-26T00:18:39.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-26T00:18:39.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:18:39.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:18:39.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-26T00:18:39.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:18:39.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:18:39.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:18:39.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:18:39.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:18:39.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:18:39.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:18:39.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:18:39.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:18:39.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:18:39.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:18:39.837 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:18:39.837 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:18:39.837 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:18:39.837 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:18:39.837 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:18:39.837 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:18:39.837 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:18:39.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:18:39.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:18:39.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:18:39.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:18:39.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:18:39.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:18:39.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:18:39.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:18:39.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:18:39.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:18:39.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:18:39.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-26T00:18:39.839 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.1 2024-03-26T00:18:39.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:18:39.839 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-26T00:18:39.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-26T00:18:39.841 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:18:39.841 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:18:39.841 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:18:39.878 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:18:39.874+0000 7f3b548c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:18:39.881 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:18:39.878+0000 7f3b548c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:18:39.882 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:18:39.878+0000 7f3b548c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:18:40.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-26T00:18:40.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:18:40.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:18:40.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:18:40.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:18:40.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:18:40.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:18:40.157 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:18:40.158 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:18:40.158 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:18:40.426 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:18:40.422+0000 7f3b548c27c0 -1 Falling back to public interface 2024-03-26T00:18:40.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:18:41.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:18:41.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:18:41.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:18:41.509 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:18:41.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:18:41.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:18:41.828 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:18:42.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:18:42.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:18:42.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:18:42.831 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:18:42.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:18:42.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:18:43.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:18:43.324 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:18:43.322+0000 7f3b548c27c0 -1 osd.1 150 log_to_monitors true 2024-03-26T00:18:44.264 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:18:44.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:18:44.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:18:44.265 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:18:44.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:18:44.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:18:44.580 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1 up in weight 1 up_from 154 up_thru 154 down_at 151 last_clean_interval [144,150) [v2:127.0.0.1:6810/53929943,v1:127.0.0.1:6811/53929943] [v2:127.0.0.1:6812/53929943,v1:127.0.0.1:6813/53929943] exists,up 0c2624cc-3bc6-4489-918a-a2155ff79e95 2024-03-26T00:18:44.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:18:44.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:18:44.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:18:44.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:18:44.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:18:44.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:18:44.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:18:44.581 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:18:44.582 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:18:44.582 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:18:44.582 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:18:44.582 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:18:44.582 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:18:44.691 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:18:44.691 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:18:44.691 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:18:44.692 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:18:44.692 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:18:44.692 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:18:44.994 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:18:44.994 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:18:44.994 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:18:44.994 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:18:44.994 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:18:45.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=639950127109 2024-03-26T00:18:45.146 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 639950127109 2024-03-26T00:18:45.146 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-639950127109' 2024-03-26T00:18:45.146 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:18:45.147 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:18:45.290 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=661424963586 2024-03-26T00:18:45.290 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 661424963586 2024-03-26T00:18:45.290 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-639950127109 1-661424963586' 2024-03-26T00:18:45.291 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:18:45.291 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-639950127109 2024-03-26T00:18:45.291 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:18:45.292 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:18:45.293 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-639950127109 2024-03-26T00:18:45.293 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:18:45.294 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=639950127109 2024-03-26T00:18:45.295 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 639950127109' 2024-03-26T00:18:45.295 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 639950127109 2024-03-26T00:18:45.295 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:18:45.597 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 639950127108 -lt 639950127109 2024-03-26T00:18:45.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:18:46.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:18:46.601 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:18:46.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 639950127110 -lt 639950127109 2024-03-26T00:18:46.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:18:46.912 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-661424963586 2024-03-26T00:18:46.912 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:18:46.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:18:46.919 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-661424963586 2024-03-26T00:18:46.919 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:18:46.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=661424963586 2024-03-26T00:18:46.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 661424963586' 2024-03-26T00:18:46.919 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 661424963586 2024-03-26T00:18:46.919 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:18:47.235 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 661424963586 -lt 661424963586 2024-03-26T00:18:47.235 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:18:47.236 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:18:47.236 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:18:47.639 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:18:47.639 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:18:47.640 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:18:47.640 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:18:47.640 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:18:47.640 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:18:47.641 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:18:47.641 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:18:47.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:18:47.965 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:18:47.965 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:18:47.965 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:18:48.324 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:18:48.324 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:18:48.324 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:18:48.324 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-26T00:18:48.324 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ17 2024-03-26T00:18:48.324 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 17 % 2 2024-03-26T00:18:48.328 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-26T00:18:48.328 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-26T00:18:48.328 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1124: TEST_corrupt_scrub_replicated: local payload=ROBJ17 2024-03-26T00:18:48.328 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1125: TEST_corrupt_scrub_replicated: echo ROBJ17 2024-03-26T00:18:48.328 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1126: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-03-26T00:18:48.328 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:18:48.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:18:48.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-26T00:18:48.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:18:48.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-03-26T00:18:48.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:18:48.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:18:48.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-26T00:18:48.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:18:48.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-26T00:18:48.329 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:18:48.329 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:18:48.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:18:48.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:18:48.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:18:48.638 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:18:48.638 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-03-26T00:18:48.638 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:18:48.638 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:18:48.638 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:18:48.639 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:18:48.639 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:18:48.639 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-03-26T00:18:49.940 INFO:tasks.workunit.client.0.smithi129.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-03-26T00:18:50.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-26T00:18:50.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:18:50.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:18:50.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-26T00:18:50.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:18:50.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:18:50.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:18:50.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:18:50.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:18:50.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:18:50.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-26T00:18:50.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-26T00:18:50.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:18:50.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:18:50.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:18:50.840 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:18:50.840 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:18:50.840 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:18:50.842 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:18:50.842 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:18:50.842 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:18:50.842 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:18:50.842 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:18:50.843 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:18:50.843 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:18:50.843 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:18:50.843 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:18:50.843 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:18:50.843 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:18:50.843 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:18:50.843 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:18:50.844 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:18:50.844 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-26T00:18:50.845 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-26T00:18:50.845 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.0 2024-03-26T00:18:50.845 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:18:50.845 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-26T00:18:50.845 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-26T00:18:50.845 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:18:50.846 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:18:50.846 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:18:50.883 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:18:50.882+0000 7fb811fca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:18:50.885 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:18:50.886+0000 7fb811fca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:18:50.887 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:18:50.886+0000 7fb811fca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:18:51.154 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-26T00:18:51.154 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:18:51.154 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-26T00:18:51.154 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:18:51.155 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:18:51.155 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:18:51.155 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:18:51.155 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:18:51.155 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:18:51.155 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:18:51.422 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:18:51.422+0000 7fb811fca7c0 -1 Falling back to public interface 2024-03-26T00:18:51.514 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:18:52.516 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:18:52.516 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:18:52.516 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:18:52.516 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:18:52.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:18:52.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:18:52.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:18:53.837 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:18:53.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:18:53.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:18:53.838 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:18:53.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:18:53.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:18:54.031 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:18:54.029+0000 7fb811fca7c0 -1 osd.0 156 log_to_monitors true 2024-03-26T00:18:54.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:18:55.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:18:55.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:18:55.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:18:55.190 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:18:55.190 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:18:55.190 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:18:55.607 INFO:tasks.workunit.client.0.smithi129.stderr:osd.0 up in weight 1 up_from 160 up_thru 160 down_at 157 last_clean_interval [149,156) [v2:127.0.0.1:6802/851505228,v1:127.0.0.1:6803/851505228] [v2:127.0.0.1:6804/851505228,v1:127.0.0.1:6805/851505228] exists,up d7a8ca59-f9ab-4120-b1c3-f87ea4b0bd36 2024-03-26T00:18:55.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:18:55.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:18:55.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:18:55.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:18:55.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:18:55.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:18:55.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:18:55.608 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:18:55.608 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:18:55.608 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:18:55.609 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:18:55.609 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:18:55.609 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:18:55.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:18:55.728 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:18:55.728 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:18:55.728 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:18:55.728 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:18:55.728 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:18:56.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:18:56.047 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:18:56.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:18:56.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:18:56.048 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:18:56.196 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=687194767362 2024-03-26T00:18:56.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 687194767362 2024-03-26T00:18:56.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-687194767362' 2024-03-26T00:18:56.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:18:56.197 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:18:56.401 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=661424963589 2024-03-26T00:18:56.401 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 661424963589 2024-03-26T00:18:56.401 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-687194767362 1-661424963589' 2024-03-26T00:18:56.401 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:18:56.402 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-687194767362 2024-03-26T00:18:56.402 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:18:56.403 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:18:56.404 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-687194767362 2024-03-26T00:18:56.404 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:18:56.405 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=687194767362 2024-03-26T00:18:56.405 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 687194767362' 2024-03-26T00:18:56.405 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 687194767362 2024-03-26T00:18:56.406 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:18:56.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 687194767362 -lt 687194767362 2024-03-26T00:18:56.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:18:56.714 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-661424963589 2024-03-26T00:18:56.714 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:18:56.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:18:56.717 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-661424963589 2024-03-26T00:18:56.717 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:18:56.718 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=661424963589 2024-03-26T00:18:56.718 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 661424963589' 2024-03-26T00:18:56.718 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 661424963589 2024-03-26T00:18:56.719 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:18:57.066 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 661424963589 -lt 661424963589 2024-03-26T00:18:57.066 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:18:57.067 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:18:57.067 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:18:57.410 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:18:57.410 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:18:57.411 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:18:57.411 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:18:57.411 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:18:57.411 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:18:57.411 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:18:57.412 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:18:57.724 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:18:57.724 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:18:57.724 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:18:57.725 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:18:58.084 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:18:58.084 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:18:58.084 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:18:58.084 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1127: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-03-26T00:18:58.084 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:18:58.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:18:58.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-26T00:18:58.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:18:58.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-03-26T00:18:58.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:18:58.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:18:58.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-26T00:18:58.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:18:58.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-26T00:18:58.085 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:18:58.085 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:18:58.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:18:58.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:18:58.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:18:58.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:18:58.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-03-26T00:18:58.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:18:58.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:18:58.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:18:58.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:18:58.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:18:58.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-03-26T00:18:59.407 INFO:tasks.workunit.client.0.smithi129.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-03-26T00:18:59.408 INFO:tasks.workunit.client.0.smithi129.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-03-26T00:18:59.980 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-26T00:18:59.981 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:18:59.981 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:18:59.981 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-26T00:18:59.981 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:18:59.981 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:18:59.981 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:18:59.981 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:18:59.981 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:18:59.981 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:18:59.981 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:18:59.981 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:18:59.982 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:18:59.982 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:18:59.982 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:18:59.984 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:18:59.985 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:18:59.985 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:18:59.985 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:18:59.985 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:18:59.985 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:18:59.985 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:18:59.985 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:18:59.985 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:18:59.985 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:18:59.985 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:18:59.985 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:18:59.985 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:18:59.985 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:18:59.985 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:18:59.985 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:18:59.986 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:18:59.986 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:18:59.986 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-26T00:18:59.986 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.1 2024-03-26T00:18:59.986 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:18:59.986 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-26T00:18:59.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-26T00:18:59.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:18:59.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:18:59.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:19:00.018 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:19:00.017+0000 7f0623cb57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:19:00.021 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:19:00.021+0000 7f0623cb57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:19:00.023 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:19:00.025+0000 7f0623cb57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:19:00.304 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-26T00:19:00.305 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:19:00.305 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:19:00.305 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:19:00.305 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:19:00.305 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:19:00.305 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:19:00.305 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:19:00.305 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:19:00.305 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:19:00.558 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:19:00.557+0000 7f0623cb57c0 -1 Falling back to public interface 2024-03-26T00:19:00.640 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:19:01.642 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:19:01.642 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:19:01.642 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:19:01.643 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:19:01.643 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:19:01.643 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:19:01.961 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:19:02.954 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:19:02.953+0000 7f0623cb57c0 -1 osd.1 161 log_to_monitors true 2024-03-26T00:19:02.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:19:02.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:19:02.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:19:02.963 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:19:02.963 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:19:02.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:19:03.422 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1 up in weight 1 up_from 165 up_thru 160 down_at 162 last_clean_interval [154,161) [v2:127.0.0.1:6810/172501519,v1:127.0.0.1:6811/172501519] [v2:127.0.0.1:6812/172501519,v1:127.0.0.1:6813/172501519] exists,up 0c2624cc-3bc6-4489-918a-a2155ff79e95 2024-03-26T00:19:03.422 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:19:03.422 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:19:03.423 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:19:03.423 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:19:03.423 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:19:03.423 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:19:03.423 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:19:03.424 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:19:03.424 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:19:03.424 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:19:03.424 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:19:03.425 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:19:03.425 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:19:03.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:19:03.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:19:03.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:19:03.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:19:03.547 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:19:03.547 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:19:04.071 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:19:04.071 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:19:04.071 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:19:04.071 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:19:04.071 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:19:04.222 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=687194767364 2024-03-26T00:19:04.222 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 687194767364 2024-03-26T00:19:04.222 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-687194767364' 2024-03-26T00:19:04.222 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:19:04.223 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:19:04.486 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=708669603842 2024-03-26T00:19:04.487 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 708669603842 2024-03-26T00:19:04.487 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-687194767364 1-708669603842' 2024-03-26T00:19:04.487 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:19:04.487 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-687194767364 2024-03-26T00:19:04.487 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:19:04.488 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:19:04.489 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-687194767364 2024-03-26T00:19:04.489 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:19:04.491 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=687194767364 2024-03-26T00:19:04.491 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 687194767364' 2024-03-26T00:19:04.491 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 687194767364 2024-03-26T00:19:04.491 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:19:04.977 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 687194767363 -lt 687194767364 2024-03-26T00:19:04.977 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:19:05.978 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:19:05.979 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:19:06.290 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 687194767365 -lt 687194767364 2024-03-26T00:19:06.290 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:19:06.291 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-708669603842 2024-03-26T00:19:06.291 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:19:06.292 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:19:06.293 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-708669603842 2024-03-26T00:19:06.293 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:19:06.295 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=708669603842 2024-03-26T00:19:06.295 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 708669603842' 2024-03-26T00:19:06.295 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 708669603842 2024-03-26T00:19:06.295 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:19:06.584 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 708669603842 -lt 708669603842 2024-03-26T00:19:06.584 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:19:06.584 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:19:06.585 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:19:06.977 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:19:06.977 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:19:06.978 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:19:06.978 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:19:06.978 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:19:06.978 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:19:06.979 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:19:06.979 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:19:07.291 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-26T00:19:07.292 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:19:07.292 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:19:07.292 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:19:07.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-26T00:19:07.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-26T00:19:07.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-26T00:19:07.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-26T00:19:07.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-26T00:19:07.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-26T00:19:07.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-26T00:19:07.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:19:07.747 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:19:07.747 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:19:07.747 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:19:07.747 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:19:07.748 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:19:07.748 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:19:08.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-26T00:19:08.048 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:19:08.048 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:19:08.048 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:19:08.429 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-26T00:19:08.429 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-26T00:19:08.429 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-26T00:19:08.429 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-26T00:19:08.429 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-26T00:19:08.430 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-26T00:19:08.430 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-26T00:19:08.431 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-26T00:19:08.431 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-26T00:19:08.792 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-26T00:19:08.792 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-26T00:19:08.792 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-26T00:19:08.792 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-26T00:19:08.792 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-26T00:19:08.993 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-26T00:19:08.993 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:19:08.994 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:19:08.994 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:19:08.994 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:19:08.995 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:19:08.995 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:19:08.995 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:19:09.290 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-26T00:19:09.291 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:19:09.291 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:19:09.291 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:19:09.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-26T00:19:09.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-26T00:19:09.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-26T00:19:09.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-26T00:19:09.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-26T00:19:09.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-26T00:19:09.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-26T00:19:09.647 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-26T00:19:09.647 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-26T00:19:10.004 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-26T00:19:10.004 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-26T00:19:10.004 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-26T00:19:10.004 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-26T00:19:10.005 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-26T00:19:10.405 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-26T00:19:10.405 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:19:10.406 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:19:10.406 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:19:10.406 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:19:10.406 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:19:10.407 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:19:10.407 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:19:10.725 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:19:10.725 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:19:10.725 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:19:10.726 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:19:11.098 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:19:11.098 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:19:11.098 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:19:11.098 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-26T00:19:11.099 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ18 2024-03-26T00:19:11.099 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 18 % 2 2024-03-26T00:19:11.101 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-26T00:19:11.101 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-26T00:19:11.101 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1132: TEST_corrupt_scrub_replicated: local payload=ROBJ18 2024-03-26T00:19:11.101 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1133: TEST_corrupt_scrub_replicated: echo ROBJ18 2024-03-26T00:19:11.101 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1134: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-03-26T00:19:11.101 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:19:11.101 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:19:11.101 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-26T00:19:11.101 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:19:11.101 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-03-26T00:19:11.101 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:19:11.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:19:11.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-26T00:19:11.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:19:11.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-26T00:19:11.102 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:19:11.102 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:19:11.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:19:11.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:19:11.103 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:19:11.411 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:19:11.411 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-03-26T00:19:11.412 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:19:11.412 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:19:11.412 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:19:11.412 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:19:11.412 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:19:11.412 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-03-26T00:19:12.939 INFO:tasks.workunit.client.0.smithi129.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-03-26T00:19:13.828 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-26T00:19:13.828 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:19:13.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:19:13.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-26T00:19:13.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:19:13.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:19:13.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:19:13.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:19:13.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:19:13.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:19:13.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-26T00:19:13.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-26T00:19:13.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:19:13.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:19:13.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:19:13.829 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:19:13.830 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:19:13.830 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:19:13.830 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:19:13.830 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:19:13.830 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:19:13.831 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:19:13.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:19:13.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:19:13.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:19:13.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:19:13.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:19:13.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:19:13.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:19:13.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:19:13.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:19:13.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:19:13.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-26T00:19:13.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-26T00:19:13.832 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.0 2024-03-26T00:19:13.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:19:13.833 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-26T00:19:13.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-26T00:19:13.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:19:13.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:19:13.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:19:13.865 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:19:13.865+0000 7f2e20c2d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:19:13.868 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:19:13.869+0000 7f2e20c2d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:19:13.869 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:19:13.869+0000 7f2e20c2d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:19:14.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-26T00:19:14.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:19:14.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-26T00:19:14.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:19:14.154 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:19:14.154 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:19:14.154 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:19:14.154 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:19:14.154 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:19:14.154 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:19:14.402 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:19:14.401+0000 7f2e20c2d7c0 -1 Falling back to public interface 2024-03-26T00:19:14.519 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:19:15.520 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:19:15.521 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:19:15.521 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:19:15.521 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:19:15.521 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:19:15.522 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:19:15.818 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:19:16.819 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:19:16.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:19:16.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:19:16.820 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:19:16.821 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:19:16.821 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:19:17.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:19:17.182 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:19:17.181+0000 7f2e20c2d7c0 -1 osd.0 167 log_to_monitors true 2024-03-26T00:19:18.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:19:18.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:19:18.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:19:18.140 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:19:18.141 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:19:18.141 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:19:18.481 INFO:tasks.workunit.client.0.smithi129.stderr:osd.0 up in weight 1 up_from 171 up_thru 166 down_at 168 last_clean_interval [160,167) [v2:127.0.0.1:6802/1212647000,v1:127.0.0.1:6803/1212647000] [v2:127.0.0.1:6804/1212647000,v1:127.0.0.1:6805/1212647000] exists,up d7a8ca59-f9ab-4120-b1c3-f87ea4b0bd36 2024-03-26T00:19:18.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:19:18.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:19:18.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:19:18.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:19:18.482 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:19:18.482 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:19:18.482 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:19:18.483 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:19:18.483 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:19:18.483 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:19:18.483 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:19:18.483 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:19:18.483 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:19:18.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:19:18.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:19:18.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:19:18.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:19:18.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:19:18.588 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:19:18.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:19:18.926 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:19:18.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:19:18.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:19:18.927 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:19:19.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=734439407618 2024-03-26T00:19:19.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 734439407618 2024-03-26T00:19:19.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-734439407618' 2024-03-26T00:19:19.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:19:19.136 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:19:19.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=708669603846 2024-03-26T00:19:19.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 708669603846 2024-03-26T00:19:19.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-734439407618 1-708669603846' 2024-03-26T00:19:19.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:19:19.278 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-734439407618 2024-03-26T00:19:19.278 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:19:19.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:19:19.281 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-734439407618 2024-03-26T00:19:19.281 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:19:19.282 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=734439407618 2024-03-26T00:19:19.282 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 734439407618' 2024-03-26T00:19:19.282 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 734439407618 2024-03-26T00:19:19.283 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:19:19.584 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 734439407618 2024-03-26T00:19:19.584 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:19:20.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:19:20.587 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:19:20.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 734439407618 -lt 734439407618 2024-03-26T00:19:20.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:19:20.928 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-708669603846 2024-03-26T00:19:20.928 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:19:20.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:19:20.932 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-708669603846 2024-03-26T00:19:20.932 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:19:20.932 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=708669603846 2024-03-26T00:19:20.933 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 708669603846' 2024-03-26T00:19:20.933 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 708669603846 2024-03-26T00:19:20.933 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:19:21.311 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 708669603846 -lt 708669603846 2024-03-26T00:19:21.311 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:19:21.312 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:19:21.312 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:19:21.690 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:19:21.691 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:19:21.692 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:19:21.692 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:19:21.692 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:19:21.692 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:19:21.692 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:19:21.692 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:19:21.992 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-26T00:19:21.992 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:19:21.992 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:19:21.993 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:19:22.369 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-26T00:19:22.369 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-26T00:19:22.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-26T00:19:22.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-26T00:19:22.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-26T00:19:22.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-26T00:19:22.470 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-26T00:19:22.471 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:19:22.472 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:19:22.472 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:19:22.472 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:19:22.472 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:19:22.472 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:19:22.473 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:19:22.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-26T00:19:22.802 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:19:22.803 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:19:22.803 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:19:23.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-26T00:19:23.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-26T00:19:23.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-26T00:19:23.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-26T00:19:23.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-26T00:19:23.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-26T00:19:23.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-26T00:19:23.183 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-26T00:19:23.183 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-26T00:19:23.538 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-26T00:19:23.538 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-26T00:19:23.538 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-26T00:19:23.538 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-26T00:19:23.538 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-26T00:19:23.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-26T00:19:23.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:19:23.742 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:19:23.742 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:19:23.742 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:19:23.742 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:19:23.742 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:19:23.743 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:19:24.054 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-26T00:19:24.054 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:19:24.055 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:19:24.055 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:19:24.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-26T00:19:24.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-26T00:19:24.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-26T00:19:24.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-26T00:19:24.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-26T00:19:24.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-26T00:19:24.418 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-26T00:19:24.419 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-26T00:19:24.419 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-26T00:19:24.787 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-26T00:19:24.787 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-26T00:19:24.787 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-26T00:19:24.787 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-26T00:19:24.787 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-26T00:19:25.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-26T00:19:25.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:19:25.189 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:19:25.189 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:19:25.189 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:19:25.189 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:19:25.190 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:19:25.190 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:19:25.501 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:19:25.501 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:19:25.502 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:19:25.502 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:19:25.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:19:25.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:19:25.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:19:25.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1135: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-03-26T00:19:25.862 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:19:25.862 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:19:25.862 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-26T00:19:25.862 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:19:25.862 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-03-26T00:19:25.862 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:19:25.862 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:19:25.862 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-26T00:19:25.862 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:19:25.862 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-26T00:19:25.862 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:19:25.862 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:19:25.862 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:19:25.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:19:25.863 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:19:27.172 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:19:27.233 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-03-26T00:19:27.233 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:19:27.233 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:19:27.233 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:19:27.234 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:19:27.234 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:19:27.234 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-03-26T00:19:27.826 INFO:tasks.workunit.client.0.smithi129.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-03-26T00:19:28.485 INFO:tasks.workunit.client.0.smithi129.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-03-26T00:19:28.492 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-26T00:19:28.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:19:28.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:19:28.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-26T00:19:28.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:19:28.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:19:28.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:19:28.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:19:28.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:19:28.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:19:28.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:19:28.495 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:19:28.495 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:19:28.495 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:19:28.495 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:19:28.495 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:19:28.496 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:19:28.496 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:19:28.496 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:19:28.496 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:19:28.496 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:19:28.496 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:19:28.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:19:28.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:19:28.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:19:28.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:19:28.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:19:28.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:19:28.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:19:28.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:19:28.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:19:28.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:19:28.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:19:28.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-26T00:19:28.497 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.1 2024-03-26T00:19:28.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:19:28.498 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-26T00:19:28.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-26T00:19:28.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:19:28.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:19:28.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:19:28.498 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:19:28.449+0000 7fca9c0557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:19:28.498 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:19:28.453+0000 7fca9c0557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:19:28.498 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:19:28.453+0000 7fca9c0557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:19:28.748 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-26T00:19:28.748 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:19:28.748 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:19:28.748 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:19:28.748 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:19:28.748 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:19:28.748 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:19:28.748 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:19:28.748 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:19:28.748 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:19:29.014 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:19:29.013+0000 7fca9c0557c0 -1 Falling back to public interface 2024-03-26T00:19:29.063 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:19:30.065 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:19:30.065 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:19:30.065 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:19:30.065 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:19:30.066 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:19:30.066 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:19:30.377 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:19:31.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:19:31.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:19:31.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:19:31.380 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:19:31.380 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:19:31.380 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:19:31.506 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:19:31.505+0000 7fca9c0557c0 -1 osd.1 173 log_to_monitors true 2024-03-26T00:19:31.812 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:19:32.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:19:32.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:19:32.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:19:32.814 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:19:32.815 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:19:32.815 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:19:33.197 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1 up in weight 1 up_from 177 up_thru 177 down_at 174 last_clean_interval [165,173) [v2:127.0.0.1:6810/2355402718,v1:127.0.0.1:6811/2355402718] [v2:127.0.0.1:6812/2355402718,v1:127.0.0.1:6813/2355402718] exists,up 0c2624cc-3bc6-4489-918a-a2155ff79e95 2024-03-26T00:19:33.198 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:19:33.198 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:19:33.198 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:19:33.198 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:19:33.198 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:19:33.198 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:19:33.198 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:19:33.199 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:19:33.199 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:19:33.199 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:19:33.199 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:19:33.199 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:19:33.199 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:19:33.316 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:19:33.316 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:19:33.317 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:19:33.317 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:19:33.317 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:19:33.317 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:19:33.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:19:33.688 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:19:33.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:19:33.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:19:33.689 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:19:33.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=734439407622 2024-03-26T00:19:33.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 734439407622 2024-03-26T00:19:33.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-734439407622' 2024-03-26T00:19:33.885 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:19:33.885 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:19:34.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=760209211394 2024-03-26T00:19:34.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 760209211394 2024-03-26T00:19:34.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-734439407622 1-760209211394' 2024-03-26T00:19:34.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:19:34.027 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-734439407622 2024-03-26T00:19:34.027 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:19:34.028 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:19:34.029 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-734439407622 2024-03-26T00:19:34.029 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:19:34.030 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=734439407622 2024-03-26T00:19:34.031 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 734439407622' 2024-03-26T00:19:34.031 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 734439407622 2024-03-26T00:19:34.031 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:19:34.325 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 734439407620 -lt 734439407622 2024-03-26T00:19:34.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:19:35.327 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:19:35.328 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:19:35.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 734439407622 -lt 734439407622 2024-03-26T00:19:35.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:19:35.652 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-760209211394 2024-03-26T00:19:35.653 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:19:35.655 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:19:35.656 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-760209211394 2024-03-26T00:19:35.656 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:19:35.657 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=760209211394 2024-03-26T00:19:35.657 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 760209211394' 2024-03-26T00:19:35.657 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 760209211394 2024-03-26T00:19:35.658 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:19:36.011 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 760209211394 -lt 760209211394 2024-03-26T00:19:36.012 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:19:36.013 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:19:36.013 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:19:36.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:19:36.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:19:36.393 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:19:36.394 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:19:36.394 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:19:36.394 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:19:36.394 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:19:36.394 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:19:36.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:19:36.728 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:19:36.728 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:19:36.728 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:19:37.101 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:19:37.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:19:37.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:19:37.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1137: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-03-26T00:19:37.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:19:37.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:19:37.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-26T00:19:37.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:19:37.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-03-26T00:19:37.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:19:37.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:19:37.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-26T00:19:37.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:19:37.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-26T00:19:37.102 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:19:37.103 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:19:37.103 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:19:37.103 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:19:37.103 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:19:37.412 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:19:37.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-03-26T00:19:37.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:19:37.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:19:37.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:19:37.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:19:37.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:19:37.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 corrupt-info 2024-03-26T00:19:38.480 INFO:tasks.workunit.client.0.smithi129.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-03-26T00:19:38.480 INFO:tasks.workunit.client.0.smithi129.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-03-26T00:19:39.031 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-26T00:19:39.031 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:19:39.031 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:19:39.031 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-26T00:19:39.031 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:19:39.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:19:39.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:19:39.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:19:39.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:19:39.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:19:39.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:19:39.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:19:39.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:19:39.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:19:39.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:19:39.034 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:19:39.035 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:19:39.035 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:19:39.035 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:19:39.035 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:19:39.035 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:19:39.035 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:19:39.035 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:19:39.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:19:39.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:19:39.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:19:39.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:19:39.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:19:39.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:19:39.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:19:39.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:19:39.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:19:39.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:19:39.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-26T00:19:39.037 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.1 2024-03-26T00:19:39.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:19:39.037 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-26T00:19:39.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-26T00:19:39.038 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:19:39.039 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:19:39.039 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:19:39.071 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:19:39.068+0000 7f14fdadd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:19:39.097 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:19:39.096+0000 7f14fdadd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:19:39.099 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:19:39.096+0000 7f14fdadd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:19:39.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-26T00:19:39.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:19:39.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:19:39.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:19:39.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:19:39.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:19:39.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:19:39.390 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:19:39.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:19:39.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:19:39.642 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:19:39.640+0000 7f14fdadd7c0 -1 Falling back to public interface 2024-03-26T00:19:39.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:19:40.728 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:19:40.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:19:40.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:19:40.729 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:19:40.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:19:40.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:19:41.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:19:41.752 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:19:41.752+0000 7f14fdadd7c0 -1 osd.1 179 log_to_monitors true 2024-03-26T00:19:42.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:19:42.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:19:42.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:19:42.048 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:19:42.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:19:42.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:19:42.676 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1 up in weight 1 up_from 183 up_thru 177 down_at 180 last_clean_interval [177,179) [v2:127.0.0.1:6810/2750693034,v1:127.0.0.1:6811/2750693034] [v2:127.0.0.1:6812/2750693034,v1:127.0.0.1:6813/2750693034] exists,up 0c2624cc-3bc6-4489-918a-a2155ff79e95 2024-03-26T00:19:42.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:19:42.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:19:42.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:19:42.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:19:42.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:19:42.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:19:42.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:19:42.677 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:19:42.677 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:19:42.677 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:19:42.677 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:19:42.677 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:19:42.678 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:19:42.799 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:19:42.799 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:19:42.799 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:19:42.799 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:19:42.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:19:42.800 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:19:43.311 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:19:43.311 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:19:43.311 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:19:43.311 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:19:43.311 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:19:43.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=734439407625 2024-03-26T00:19:43.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 734439407625 2024-03-26T00:19:43.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-734439407625' 2024-03-26T00:19:43.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:19:43.568 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:19:43.813 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=785979015170 2024-03-26T00:19:43.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 785979015170 2024-03-26T00:19:43.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-734439407625 1-785979015170' 2024-03-26T00:19:43.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:19:43.814 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-734439407625 2024-03-26T00:19:43.814 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:19:43.815 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:19:43.816 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-734439407625 2024-03-26T00:19:43.816 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:19:43.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=734439407625 2024-03-26T00:19:43.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 734439407625' 2024-03-26T00:19:43.817 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 734439407625 2024-03-26T00:19:43.818 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:19:44.129 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 734439407623 -lt 734439407625 2024-03-26T00:19:44.129 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:19:45.130 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:19:45.131 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:19:45.455 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 734439407625 -lt 734439407625 2024-03-26T00:19:45.455 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:19:45.456 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-785979015170 2024-03-26T00:19:45.456 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:19:45.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:19:45.460 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-785979015170 2024-03-26T00:19:45.460 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:19:45.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=785979015170 2024-03-26T00:19:45.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 785979015170' 2024-03-26T00:19:45.461 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 785979015170 2024-03-26T00:19:45.462 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:19:45.774 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 785979015170 -lt 785979015170 2024-03-26T00:19:45.775 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:19:45.775 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:19:45.776 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:19:46.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:19:46.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:19:46.124 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:19:46.124 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:19:46.124 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:19:46.124 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:19:46.125 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:19:46.125 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:19:46.469 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-26T00:19:46.470 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:19:46.470 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:19:46.471 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:19:46.849 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-26T00:19:46.849 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-26T00:19:46.849 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-26T00:19:46.849 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-26T00:19:46.849 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-26T00:19:46.849 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-26T00:19:46.950 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-26T00:19:46.950 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:19:46.951 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:19:46.951 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:19:46.951 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:19:46.952 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:19:46.952 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:19:46.952 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:19:47.292 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-26T00:19:47.293 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:19:47.293 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:19:47.293 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:19:47.675 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-26T00:19:47.675 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-26T00:19:47.675 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-26T00:19:47.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-26T00:19:47.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-26T00:19:47.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-26T00:19:47.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-26T00:19:47.676 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-26T00:19:47.676 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-26T00:19:48.096 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-26T00:19:48.096 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-26T00:19:48.096 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-26T00:19:48.096 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-26T00:19:48.096 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-26T00:19:48.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-26T00:19:48.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:19:48.298 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:19:48.298 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:19:48.299 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:19:48.299 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:19:48.299 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:19:48.299 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:19:48.593 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:19:48.593 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:19:48.594 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:19:48.594 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:19:48.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:19:48.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:19:48.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:19:48.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-26T00:19:48.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ19 2024-03-26T00:19:48.950 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 19 % 2 2024-03-26T00:19:48.951 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-26T00:19:48.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-26T00:19:48.952 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1146: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2024-03-26T00:19:48.952 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=csr_pool 2024-03-26T00:19:48.952 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=ROBJ0 2024-03-26T00:19:48.953 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-03-26T00:19:48.953 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-26T00:19:49.308 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1146: TEST_corrupt_scrub_replicated: local pg=3.0 2024-03-26T00:19:49.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1148: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=1048576 2024-03-26T00:19:49.440 INFO:tasks.workunit.client.0.smithi129.stdout:osd.0: {} 2024-03-26T00:19:49.441 INFO:tasks.workunit.client.0.smithi129.stderr:osd.0: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_max_object_size = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-03-26T00:19:49.457 INFO:tasks.workunit.client.0.smithi129.stdout:osd.1: {} 2024-03-26T00:19:49.457 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_max_object_size = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-03-26T00:19:49.472 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1150: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2024-03-26T00:19:49.472 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-03-26T00:19:49.472 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-26T00:19:49.472 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-03-26T00:19:49.472 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-26T00:19:49.472 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-03-26T00:19:49.473 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-26T00:19:49.473 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ11 2024-03-26T00:19:49.473 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-26T00:19:49.473 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-03-26T00:19:49.473 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-26T00:19:49.473 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-03-26T00:19:49.473 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-26T00:19:49.473 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ11 2024-03-26T00:19:49.473 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-03-26T00:19:49.473 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ11 2024-03-26T00:19:49.474 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ11 2024-03-26T00:19:49.474 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-26T00:19:49.777 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-03-26T00:19:49.777 INFO:tasks.workunit.client.0.smithi129.stderr:0' 2024-03-26T00:19:49.777 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-03-26T00:19:49.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-03-26T00:19:49.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-26T00:19:49.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-03-26T00:19:49.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-03-26T00:19:49.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-03-26T00:19:49.778 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-03-26T00:19:49.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-26T00:19:49.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-03-26T00:19:49.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-26T00:19:49.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-03-26T00:19:49.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-26T00:19:49.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-26T00:19:49.780 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-26T00:19:49.780 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-03-26T00:19:49.780 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-26T00:19:49.780 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-26T00:19:49.781 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-26T00:19:49.781 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:19:49.781 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:19:49.782 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-osd.1.asok 2024-03-26T00:19:49.782 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61636/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-03-26T00:19:49.889 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-26T00:19:49.889 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-26T00:19:49.892 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-03-26T00:19:49.892 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-26T00:19:49.892 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-26T00:19:49.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-26T00:19:49.892 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-26T00:19:49.892 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:19:49.892 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:19:49.892 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-osd.1.asok 2024-03-26T00:19:49.893 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-26T00:19:49.893 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.61636/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2024-03-26T00:19:50.012 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1151: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2024-03-26T00:19:50.012 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-03-26T00:19:50.012 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-26T00:19:50.012 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-03-26T00:19:50.012 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-26T00:19:50.012 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-03-26T00:19:50.012 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-26T00:19:50.013 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ12 2024-03-26T00:19:50.013 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-26T00:19:50.013 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-03-26T00:19:50.013 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-26T00:19:50.013 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-03-26T00:19:50.013 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-26T00:19:50.013 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ12 2024-03-26T00:19:50.013 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-03-26T00:19:50.013 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ12 2024-03-26T00:19:50.013 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ12 2024-03-26T00:19:50.013 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-26T00:19:50.302 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-03-26T00:19:50.302 INFO:tasks.workunit.client.0.smithi129.stderr:0' 2024-03-26T00:19:50.302 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-03-26T00:19:50.302 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-03-26T00:19:50.302 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-26T00:19:50.302 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-03-26T00:19:50.302 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-03-26T00:19:50.302 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-03-26T00:19:50.303 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-03-26T00:19:50.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-26T00:19:50.304 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-03-26T00:19:50.304 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-26T00:19:50.304 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-03-26T00:19:50.304 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-26T00:19:50.304 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-26T00:19:50.305 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-26T00:19:50.305 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-03-26T00:19:50.305 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-26T00:19:50.305 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-26T00:19:50.306 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-26T00:19:50.306 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:19:50.306 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:19:50.306 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-osd.0.asok 2024-03-26T00:19:50.307 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61636/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-03-26T00:19:50.414 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-26T00:19:50.414 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-26T00:19:50.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-26T00:19:50.418 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-03-26T00:19:50.418 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-26T00:19:50.418 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-26T00:19:50.419 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-26T00:19:50.419 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:19:50.419 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:19:50.421 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-osd.0.asok 2024-03-26T00:19:50.422 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-26T00:19:50.422 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.61636/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2024-03-26T00:19:50.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1152: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2024-03-26T00:19:50.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-03-26T00:19:50.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-26T00:19:50.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-03-26T00:19:50.533 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-26T00:19:50.533 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-03-26T00:19:50.533 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-26T00:19:50.533 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-03-26T00:19:50.533 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-26T00:19:50.533 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-03-26T00:19:50.533 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-26T00:19:50.533 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-03-26T00:19:50.533 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-26T00:19:50.534 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-03-26T00:19:50.534 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-03-26T00:19:50.534 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-03-26T00:19:50.534 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-26T00:19:50.534 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-03-26T00:19:50.836 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-03-26T00:19:50.836 INFO:tasks.workunit.client.0.smithi129.stderr:0' 2024-03-26T00:19:50.836 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-03-26T00:19:50.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-03-26T00:19:50.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-26T00:19:50.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-03-26T00:19:50.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-03-26T00:19:50.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-03-26T00:19:50.836 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-03-26T00:19:50.837 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-26T00:19:50.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-03-26T00:19:50.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-26T00:19:50.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-03-26T00:19:50.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-26T00:19:50.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-26T00:19:50.839 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-26T00:19:50.839 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-03-26T00:19:50.839 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-26T00:19:50.839 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-26T00:19:50.839 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-26T00:19:50.840 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:19:50.840 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:19:50.840 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-osd.0.asok 2024-03-26T00:19:50.840 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61636/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-03-26T00:19:50.947 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-26T00:19:50.947 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-26T00:19:50.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-26T00:19:50.949 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-03-26T00:19:50.949 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-26T00:19:50.949 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-26T00:19:50.949 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-26T00:19:50.950 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:19:50.950 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:19:50.950 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-osd.0.asok 2024-03-26T00:19:50.950 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-26T00:19:50.950 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.61636/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2024-03-26T00:19:51.066 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1153: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2024-03-26T00:19:51.066 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-03-26T00:19:51.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-26T00:19:51.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-03-26T00:19:51.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-26T00:19:51.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-03-26T00:19:51.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-26T00:19:51.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-03-26T00:19:51.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-26T00:19:51.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-03-26T00:19:51.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-26T00:19:51.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-03-26T00:19:51.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-26T00:19:51.068 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-03-26T00:19:51.068 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-03-26T00:19:51.068 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-03-26T00:19:51.069 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-03-26T00:19:51.069 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-26T00:19:51.364 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-03-26T00:19:51.364 INFO:tasks.workunit.client.0.smithi129.stderr:0' 2024-03-26T00:19:51.364 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-03-26T00:19:51.364 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-03-26T00:19:51.364 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-26T00:19:51.365 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-03-26T00:19:51.365 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-03-26T00:19:51.365 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-03-26T00:19:51.365 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-03-26T00:19:51.366 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-26T00:19:51.366 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-03-26T00:19:51.366 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-26T00:19:51.367 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-03-26T00:19:51.367 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-26T00:19:51.367 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-26T00:19:51.367 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-26T00:19:51.368 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-03-26T00:19:51.368 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-26T00:19:51.368 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-26T00:19:51.369 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-26T00:19:51.369 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:19:51.369 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:19:51.370 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-osd.1.asok 2024-03-26T00:19:51.370 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61636/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-03-26T00:19:51.490 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-26T00:19:51.490 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-26T00:19:51.491 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-26T00:19:51.491 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-03-26T00:19:51.492 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-26T00:19:51.492 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-26T00:19:51.492 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-26T00:19:51.492 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:19:51.492 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:19:51.493 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-osd.1.asok 2024-03-26T00:19:51.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-26T00:19:51.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.61636/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2024-03-26T00:19:51.610 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1155: TEST_corrupt_scrub_replicated: pg_scrub 3.0 2024-03-26T00:19:51.610 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-03-26T00:19:51.611 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-03-26T00:19:51.611 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:19:51.611 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:19:51.612 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-26T00:19:51.612 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:19:51.904 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-03-26T00:15:16.083704+0000 2024-03-26T00:19:51.904 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-03-26T00:19:52.182 INFO:tasks.workunit.client.0.smithi129.stderr:instructing pg 3.0 on osd.1 to scrub 2024-03-26T00:19:52.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-03-26T00:15:16.083704+0000 2024-03-26T00:19:52.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-26T00:19:52.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-26T00:15:16.083704+0000 2024-03-26T00:19:52.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-26T00:19:52.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-26T00:19:52.199 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:19:52.199 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-26T00:19:52.199 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:19:52.200 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:19:52.200 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:19:52.200 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-26T00:19:52.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:15:16.083704+0000 '>' 2024-03-26T00:15:16.083704+0000 2024-03-26T00:19:52.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:19:53.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:19:53.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:19:53.512 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-26T00:19:53.512 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:19:53.512 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:19:53.513 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-26T00:19:53.513 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:19:53.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:15:16.083704+0000 '>' 2024-03-26T00:15:16.083704+0000 2024-03-26T00:19:53.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:19:54.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:19:54.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:19:54.837 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-26T00:19:54.837 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:19:54.837 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:19:54.838 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:19:54.838 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-26T00:19:55.156 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:15:16.083704+0000 '>' 2024-03-26T00:15:16.083704+0000 2024-03-26T00:19:55.156 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:19:56.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:19:56.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:19:56.158 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-26T00:19:56.158 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:19:56.159 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:19:56.159 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:19:56.159 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-26T00:19:56.475 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:15:16.083704+0000 '>' 2024-03-26T00:15:16.083704+0000 2024-03-26T00:19:56.476 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:19:57.478 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:19:57.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:19:57.607 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-26T00:19:57.608 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:19:57.608 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:19:57.608 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:19:57.609 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-26T00:19:57.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:15:16.083704+0000 '>' 2024-03-26T00:15:16.083704+0000 2024-03-26T00:19:57.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:19:58.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:19:58.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:19:58.784 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-26T00:19:58.784 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:19:58.784 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:19:58.785 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:19:58.785 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-26T00:19:59.107 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:19:52.782680+0000 '>' 2024-03-26T00:15:16.083704+0000 2024-03-26T00:19:59.107 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-26T00:19:59.107 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1157: TEST_corrupt_scrub_replicated: ERRORS=0 2024-03-26T00:19:59.107 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1158: TEST_corrupt_scrub_replicated: declare -a s_err_strings 2024-03-26T00:19:59.107 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1159: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2024-03-26T00:19:59.107 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1160: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' 2024-03-26T00:19:59.107 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1161: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2024-03-26T00:19:59.108 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1162: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2024-03-26T00:19:59.108 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1163: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2024-03-26T00:19:59.108 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1164: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2024-03-26T00:19:59.108 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1165: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2024-03-26T00:19:59.108 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1166: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2024-03-26T00:19:59.108 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1167: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2024-03-26T00:19:59.108 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1168: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2024-03-26T00:19:59.108 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1169: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2024-03-26T00:19:59.108 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1170: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2024-03-26T00:19:59.108 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1171: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2024-03-26T00:19:59.108 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1172: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2024-03-26T00:19:59.108 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1173: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' 2024-03-26T00:19:59.108 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1174: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2024-03-26T00:19:59.108 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1175: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2024-03-26T00:19:59.109 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1176: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' 2024-03-26T00:19:59.109 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1177: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-03-26T00:19:59.109 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1178: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' 2024-03-26T00:19:59.109 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1179: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' 2024-03-26T00:19:59.109 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1180: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' 2024-03-26T00:19:59.109 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:19:59.109 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-03-26T00:19:59.117 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:19:59.118 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-03-26T00:19:59.131 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:19:59.131 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2024-03-26T00:19:59.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:19:59.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2024-03-26T00:19:59.151 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:19:59.151 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-03-26T00:19:59.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:19:59.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2024-03-26T00:19:59.170 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:19:59.170 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-03-26T00:19:59.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:19:59.180 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-03-26T00:19:59.190 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:19:59.191 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2024-03-26T00:19:59.200 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:19:59.200 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-03-26T00:19:59.211 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:19:59.211 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-03-26T00:19:59.221 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:19:59.221 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2024-03-26T00:19:59.233 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:19:59.233 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-03-26T00:19:59.244 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:19:59.244 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2024-03-26T00:19:59.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:19:59.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-03-26T00:19:59.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:19:59.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-03-26T00:19:59.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:19:59.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2024-03-26T00:19:59.292 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:19:59.292 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' td/osd-scrub-repair/osd.1.log 2024-03-26T00:19:59.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:19:59.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-03-26T00:19:59.313 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:19:59.313 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-03-26T00:19:59.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:19:59.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' td/osd-scrub-repair/osd.1.log 2024-03-26T00:19:59.340 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:19:59.340 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' td/osd-scrub-repair/osd.1.log 2024-03-26T00:19:59.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1191: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2024-03-26T00:19:59.384 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1193: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2024-03-26T00:19:59.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1193: TEST_corrupt_scrub_replicated: test 1 = 1 2024-03-26T00:19:59.415 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2024-03-26T00:19:59.444 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2024-03-26T00:19:59.444 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1197: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2024-03-26T00:19:59.473 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2024-03-26T00:19:59.502 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: epoch=183 2024-03-26T00:19:59.503 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-03-26T00:19:59.503 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-03-26T00:19:59.503 INFO:tasks.workunit.client.0.smithi129.stderr: . as $in 2024-03-26T00:19:59.503 INFO:tasks.workunit.client.0.smithi129.stderr: | if type == "object" then 2024-03-26T00:19:59.503 INFO:tasks.workunit.client.0.smithi129.stderr: reduce keys[] as $key 2024-03-26T00:19:59.503 INFO:tasks.workunit.client.0.smithi129.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-26T00:19:59.503 INFO:tasks.workunit.client.0.smithi129.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-26T00:19:59.504 INFO:tasks.workunit.client.0.smithi129.stderr: else f 2024-03-26T00:19:59.504 INFO:tasks.workunit.client.0.smithi129.stderr: end; 2024-03-26T00:19:59.504 INFO:tasks.workunit.client.0.smithi129.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-26T00:19:59.504 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-26T00:19:59.504 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-26T00:19:59.504 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-26T00:19:59.504 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-03-26T00:19:59.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-26T00:19:59.551 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-03-26T00:19:59.551 INFO:tasks.workunit.client.0.smithi129.stderr: . as $in 2024-03-26T00:19:59.551 INFO:tasks.workunit.client.0.smithi129.stderr: | if type == "object" then 2024-03-26T00:19:59.551 INFO:tasks.workunit.client.0.smithi129.stderr: reduce keys[] as $key 2024-03-26T00:19:59.551 INFO:tasks.workunit.client.0.smithi129.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-26T00:19:59.551 INFO:tasks.workunit.client.0.smithi129.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-26T00:19:59.551 INFO:tasks.workunit.client.0.smithi129.stderr: else f 2024-03-26T00:19:59.551 INFO:tasks.workunit.client.0.smithi129.stderr: end; 2024-03-26T00:19:59.551 INFO:tasks.workunit.client.0.smithi129.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-26T00:19:59.551 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-26T00:19:59.552 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-26T00:19:59.552 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-26T00:19:59.552 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-03-26T00:19:59.552 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-03-26T00:19:59.552 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-26T00:19:59.597 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2083: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-26T00:19:59.597 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-26T00:19:59.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2084: TEST_corrupt_scrub_replicated: test no = yes 2024-03-26T00:19:59.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2089: TEST_corrupt_scrub_replicated: test '' = yes 2024-03-26T00:19:59.600 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2094: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-03-26T00:19:59.600 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2096: TEST_corrupt_scrub_replicated: echo -n ZZZ 2024-03-26T00:19:59.600 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2097: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2024-03-26T00:19:59.679 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2099: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-03-26T00:19:59.679 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:19:59.679 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:19:59.679 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-26T00:19:59.679 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:19:59.679 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-03-26T00:19:59.680 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:19:59.680 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:19:59.680 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-26T00:19:59.680 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:19:59.680 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-26T00:19:59.680 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:19:59.680 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:19:59.681 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:19:59.681 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:19:59.681 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:19:59.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:19:59.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-03-26T00:19:59.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:19:59.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:19:59.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:19:59.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:19:59.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:19:59.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-03-26T00:20:01.547 INFO:tasks.workunit.client.0.smithi129.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-03-26T00:20:02.091 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-26T00:20:02.091 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:20:02.092 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:20:02.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-26T00:20:02.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:20:02.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:20:02.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:20:02.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:20:02.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:20:02.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:20:02.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-26T00:20:02.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-26T00:20:02.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:20:02.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:20:02.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:20:02.095 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:20:02.096 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:20:02.096 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:20:02.096 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:20:02.096 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:20:02.096 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:20:02.096 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:20:02.096 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:20:02.096 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:20:02.096 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:20:02.096 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:20:02.096 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:20:02.096 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:20:02.096 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:20:02.096 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:20:02.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:20:02.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:20:02.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-26T00:20:02.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-26T00:20:02.097 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.0 2024-03-26T00:20:02.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:20:02.097 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-26T00:20:02.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-26T00:20:02.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:20:02.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:20:02.098 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:20:02.123 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:20:02.120+0000 7f520cbfc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:20:02.124 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:20:02.124+0000 7f520cbfc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:20:02.126 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:20:02.124+0000 7f520cbfc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:20:02.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-26T00:20:02.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:20:02.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-26T00:20:02.414 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:20:02.414 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:20:02.414 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:20:02.414 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:20:02.414 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:20:02.414 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:20:02.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:20:02.658 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:20:02.656+0000 7f520cbfc7c0 -1 Falling back to public interface 2024-03-26T00:20:02.728 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:20:03.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:20:03.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:20:03.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:20:03.730 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:20:03.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:20:03.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:20:04.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:20:05.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:20:05.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:20:05.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:20:05.046 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:20:05.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:20:05.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:20:05.362 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:20:05.479 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:20:05.456+0000 7f520cbfc7c0 -1 osd.0 185 log_to_monitors true 2024-03-26T00:20:06.365 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:20:06.365 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:20:06.365 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:20:06.365 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:20:06.365 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:20:06.366 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:20:06.702 INFO:tasks.workunit.client.0.smithi129.stderr:osd.0 up in weight 1 up_from 189 up_thru 183 down_at 186 last_clean_interval [171,185) [v2:127.0.0.1:6802/2707606247,v1:127.0.0.1:6803/2707606247] [v2:127.0.0.1:6804/2707606247,v1:127.0.0.1:6805/2707606247] exists,up d7a8ca59-f9ab-4120-b1c3-f87ea4b0bd36 2024-03-26T00:20:06.702 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:20:06.702 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:20:06.702 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:20:06.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:20:06.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:20:06.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:20:06.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:20:06.704 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:20:06.704 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:20:06.704 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:20:06.704 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:20:06.704 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:20:06.704 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:20:06.812 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:20:06.812 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:20:06.812 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:20:06.812 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:20:06.812 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:20:06.813 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:20:07.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:20:07.229 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:20:07.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:20:07.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:20:07.229 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:20:07.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=811748818946 2024-03-26T00:20:07.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 811748818946 2024-03-26T00:20:07.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-811748818946' 2024-03-26T00:20:07.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:20:07.388 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:20:07.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=785979015176 2024-03-26T00:20:07.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 785979015176 2024-03-26T00:20:07.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-811748818946 1-785979015176' 2024-03-26T00:20:07.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:20:07.527 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-811748818946 2024-03-26T00:20:07.528 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:20:07.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:20:07.529 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-811748818946 2024-03-26T00:20:07.529 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:20:07.531 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=811748818946 2024-03-26T00:20:07.531 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 811748818946' 2024-03-26T00:20:07.531 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 811748818946 2024-03-26T00:20:07.531 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:20:07.828 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 811748818946 2024-03-26T00:20:07.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:20:08.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:20:08.830 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:20:09.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 811748818946 -lt 811748818946 2024-03-26T00:20:09.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:20:09.141 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-785979015176 2024-03-26T00:20:09.141 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:20:09.143 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:20:09.144 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-785979015176 2024-03-26T00:20:09.144 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:20:09.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=785979015176 2024-03-26T00:20:09.146 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 785979015176' 2024-03-26T00:20:09.146 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 785979015176 2024-03-26T00:20:09.146 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:20:09.476 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 785979015176 -lt 785979015176 2024-03-26T00:20:09.476 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:20:09.477 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:20:09.477 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:20:09.853 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:20:09.854 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:20:09.854 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:20:09.854 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:20:09.855 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:20:09.855 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:20:09.855 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:20:09.855 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:20:10.144 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-26T00:20:10.145 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:20:10.145 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:20:10.145 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:20:10.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-26T00:20:10.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-26T00:20:10.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-26T00:20:10.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-26T00:20:10.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-26T00:20:10.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-26T00:20:10.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-26T00:20:10.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:20:10.601 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:20:10.602 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:20:10.602 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:20:10.602 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:20:10.602 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:20:10.602 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:20:10.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-26T00:20:10.915 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:20:10.915 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:20:10.915 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:20:11.273 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-26T00:20:11.274 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-26T00:20:11.274 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-26T00:20:11.274 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-26T00:20:11.274 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-26T00:20:11.274 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-26T00:20:11.274 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-26T00:20:11.275 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-26T00:20:11.275 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-26T00:20:11.621 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-26T00:20:11.621 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-26T00:20:11.621 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-26T00:20:11.621 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-26T00:20:11.621 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-26T00:20:11.823 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-26T00:20:11.823 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:20:11.824 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:20:11.824 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:20:11.825 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:20:11.825 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:20:11.825 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:20:11.825 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:20:12.131 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-26T00:20:12.132 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:20:12.132 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:20:12.133 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:20:12.475 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-26T00:20:12.476 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-26T00:20:12.476 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-26T00:20:12.476 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-26T00:20:12.476 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-26T00:20:12.476 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-26T00:20:12.476 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-26T00:20:12.476 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-26T00:20:12.477 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-26T00:20:12.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-26T00:20:12.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-26T00:20:12.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-26T00:20:12.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-26T00:20:12.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-26T00:20:13.234 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-26T00:20:13.234 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:20:13.235 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:20:13.236 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:20:13.236 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:20:13.236 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:20:13.236 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:20:13.236 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:20:13.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:20:13.550 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:20:13.551 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:20:13.551 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:20:13.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:20:13.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:20:13.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:20:13.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2100: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2024-03-26T00:20:13.909 INFO:tasks.workunit.client.0.smithi129.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2024-03-26T00:20:13.909 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2102: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-03-26T00:20:13.909 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2103: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-03-26T00:20:13.909 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:20:13.909 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:20:13.909 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-26T00:20:13.909 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:20:13.909 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-03-26T00:20:13.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:20:13.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:20:13.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-26T00:20:13.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:20:13.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-26T00:20:13.910 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:20:13.910 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:20:13.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:20:13.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:20:13.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:20:15.221 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:20:15.221 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-03-26T00:20:15.221 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:20:15.221 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:20:15.221 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:20:15.221 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:20:15.221 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:20:15.221 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 get-attr _ 2024-03-26T00:20:15.877 INFO:tasks.workunit.client.0.smithi129.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-03-26T00:20:15.877 INFO:tasks.workunit.client.0.smithi129.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-03-26T00:20:16.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-26T00:20:16.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:20:16.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:20:16.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-26T00:20:16.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:20:16.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:20:16.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:20:16.416 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:20:16.416 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:20:16.416 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:20:16.416 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:20:16.416 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:20:16.416 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:20:16.416 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:20:16.416 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:20:16.417 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:20:16.417 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:20:16.417 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:20:16.417 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:20:16.417 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:20:16.417 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:20:16.417 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:20:16.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:20:16.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:20:16.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:20:16.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:20:16.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:20:16.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:20:16.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:20:16.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:20:16.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:20:16.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:20:16.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:20:16.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-26T00:20:16.420 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.1 2024-03-26T00:20:16.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:20:16.421 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-26T00:20:16.422 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-26T00:20:16.423 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:20:16.423 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:20:16.423 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:20:16.455 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:20:16.452+0000 7f1d9f61e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:20:16.459 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:20:16.456+0000 7f1d9f61e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:20:16.460 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:20:16.460+0000 7f1d9f61e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:20:16.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-26T00:20:16.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:20:16.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:20:16.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:20:16.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:20:16.768 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:20:16.768 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:20:16.768 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:20:16.768 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:20:16.768 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:20:16.998 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:20:16.995+0000 7f1d9f61e7c0 -1 Falling back to public interface 2024-03-26T00:20:17.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:20:18.083 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:20:18.083 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:20:18.083 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:20:18.083 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:20:18.083 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:20:18.084 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:20:18.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:20:19.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:20:19.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:20:19.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:20:19.391 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:20:19.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:20:19.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:20:19.477 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:20:19.475+0000 7f1d9f61e7c0 -1 osd.1 191 log_to_monitors true 2024-03-26T00:20:19.852 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:20:20.854 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:20:20.855 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:20:20.855 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:20:20.855 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:20:20.855 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:20:20.855 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:20:21.361 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1 up in weight 1 up_from 195 up_thru 195 down_at 192 last_clean_interval [183,191) [v2:127.0.0.1:6810/1474741402,v1:127.0.0.1:6811/1474741402] [v2:127.0.0.1:6812/1474741402,v1:127.0.0.1:6813/1474741402] exists,up 0c2624cc-3bc6-4489-918a-a2155ff79e95 2024-03-26T00:20:21.361 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:20:21.361 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:20:21.361 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:20:21.361 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:20:21.361 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:20:21.361 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:20:21.361 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:20:21.362 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:20:21.363 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:20:21.363 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:20:21.364 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:20:21.364 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:20:21.364 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:20:21.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:20:21.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:20:21.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:20:21.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:20:21.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:20:21.490 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:20:21.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:20:21.893 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:20:21.893 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:20:21.893 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:20:21.893 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:20:22.091 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=811748818950 2024-03-26T00:20:22.091 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 811748818950 2024-03-26T00:20:22.091 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-811748818950' 2024-03-26T00:20:22.091 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:20:22.092 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:20:22.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=837518622722 2024-03-26T00:20:22.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 837518622722 2024-03-26T00:20:22.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-811748818950 1-837518622722' 2024-03-26T00:20:22.241 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:20:22.241 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-811748818950 2024-03-26T00:20:22.241 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:20:22.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:20:22.244 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-811748818950 2024-03-26T00:20:22.244 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:20:22.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=811748818950 2024-03-26T00:20:22.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 811748818950' 2024-03-26T00:20:22.246 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 811748818950 2024-03-26T00:20:22.246 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:20:22.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 811748818948 -lt 811748818950 2024-03-26T00:20:22.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:20:23.552 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:20:23.553 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:20:23.856 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 811748818950 -lt 811748818950 2024-03-26T00:20:23.857 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:20:23.858 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-837518622722 2024-03-26T00:20:23.858 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:20:23.859 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:20:23.861 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-837518622722 2024-03-26T00:20:23.861 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:20:23.862 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=837518622722 2024-03-26T00:20:23.862 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 837518622722' 2024-03-26T00:20:23.862 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 837518622722 2024-03-26T00:20:23.862 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:20:24.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 837518622722 -lt 837518622722 2024-03-26T00:20:24.217 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:20:24.218 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:20:24.218 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:20:24.592 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:20:24.593 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:20:24.593 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:20:24.593 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:20:24.593 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:20:24.593 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:20:24.594 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:20:24.594 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:20:24.916 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:20:24.917 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:20:24.917 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:20:24.917 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:20:25.294 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:20:25.294 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:20:25.294 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:20:25.294 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2104: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key2-ROBJ10 val2-ROBJ10 2024-03-26T00:20:25.470 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2105: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-03-26T00:20:25.470 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:20:25.471 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:20:25.471 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-26T00:20:25.471 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:20:25.471 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-03-26T00:20:25.471 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:20:25.471 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:20:25.471 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-26T00:20:25.471 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:20:25.471 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-26T00:20:25.472 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:20:25.472 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:20:25.472 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:20:25.472 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:20:25.472 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:20:26.786 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:20:26.787 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-03-26T00:20:26.787 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:20:26.787 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:20:26.787 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:20:26.787 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:20:26.787 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:20:26.787 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-03-26T00:20:28.181 INFO:tasks.workunit.client.0.smithi129.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-03-26T00:20:28.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-26T00:20:28.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:20:28.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:20:28.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-26T00:20:28.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:20:28.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:20:28.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:20:28.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:20:28.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:20:28.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:20:28.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-26T00:20:28.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-26T00:20:28.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:20:28.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:20:28.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:20:28.733 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:20:28.733 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:20:28.733 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:20:28.733 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:20:28.733 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:20:28.734 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:20:28.734 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:20:28.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:20:28.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:20:28.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:20:28.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:20:28.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:20:28.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:20:28.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:20:28.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:20:28.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:20:28.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:20:28.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-26T00:20:28.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-26T00:20:28.735 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.0 2024-03-26T00:20:28.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:20:28.736 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-26T00:20:28.737 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-26T00:20:28.737 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:20:28.737 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:20:28.738 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:20:28.769 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:20:28.767+0000 7f01219bd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:20:28.771 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:20:28.767+0000 7f01219bd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:20:28.773 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:20:28.771+0000 7f01219bd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:20:29.061 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-26T00:20:29.061 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:20:29.061 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-26T00:20:29.061 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:20:29.061 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:20:29.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:20:29.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:20:29.062 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:20:29.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:20:29.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:20:29.314 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:20:29.311+0000 7f01219bd7c0 -1 Falling back to public interface 2024-03-26T00:20:29.384 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:20:30.385 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:20:30.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:20:30.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:20:30.386 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:20:30.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:20:30.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:20:30.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:20:31.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:20:31.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:20:31.706 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:20:31.706 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:20:31.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:20:31.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:20:32.026 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:20:32.064 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:20:32.063+0000 7f01219bd7c0 -1 osd.0 197 log_to_monitors true 2024-03-26T00:20:33.028 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:20:33.028 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:20:33.029 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:20:33.029 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:20:33.029 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:20:33.029 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:20:33.496 INFO:tasks.workunit.client.0.smithi129.stderr:osd.0 up in weight 1 up_from 201 up_thru 196 down_at 198 last_clean_interval [189,197) [v2:127.0.0.1:6802/3235574496,v1:127.0.0.1:6803/3235574496] [v2:127.0.0.1:6804/3235574496,v1:127.0.0.1:6805/3235574496] exists,up d7a8ca59-f9ab-4120-b1c3-f87ea4b0bd36 2024-03-26T00:20:33.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:20:33.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:20:33.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:20:33.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:20:33.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:20:33.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:20:33.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:20:33.497 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:20:33.498 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:20:33.499 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:20:33.499 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:20:33.499 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:20:33.499 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:20:33.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:20:33.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:20:33.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:20:33.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:20:33.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:20:33.608 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:20:33.971 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:20:33.971 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:20:33.971 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:20:33.971 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:20:33.972 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:20:34.248 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=863288426498 2024-03-26T00:20:34.248 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 863288426498 2024-03-26T00:20:34.248 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-863288426498' 2024-03-26T00:20:34.249 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:20:34.249 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:20:34.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=837518622725 2024-03-26T00:20:34.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 837518622725 2024-03-26T00:20:34.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-863288426498 1-837518622725' 2024-03-26T00:20:34.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:20:34.420 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-863288426498 2024-03-26T00:20:34.420 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:20:34.421 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:20:34.423 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-863288426498 2024-03-26T00:20:34.423 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:20:34.428 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=863288426498 2024-03-26T00:20:34.428 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 863288426498' 2024-03-26T00:20:34.429 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 863288426498 2024-03-26T00:20:34.429 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:20:34.801 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 863288426498 -lt 863288426498 2024-03-26T00:20:34.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:20:34.803 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-837518622725 2024-03-26T00:20:34.803 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:20:34.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:20:34.807 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-837518622725 2024-03-26T00:20:34.807 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:20:34.807 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=837518622725 2024-03-26T00:20:34.807 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 837518622725' 2024-03-26T00:20:34.807 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 837518622725 2024-03-26T00:20:34.808 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:20:35.137 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 837518622725 -lt 837518622725 2024-03-26T00:20:35.138 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:20:35.138 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:20:35.138 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:20:35.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:20:35.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:20:35.565 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:20:35.565 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:20:35.565 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:20:35.565 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:20:35.566 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:20:35.566 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:20:35.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-26T00:20:35.908 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:20:35.909 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:20:35.909 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:20:36.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-03-26T00:20:36.258 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' -1 2024-03-26T00:20:36.258 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-26T00:20:36.258 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=5 2024-03-26T00:20:36.258 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-26T00:20:36.258 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-26T00:20:36.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-26T00:20:36.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:20:36.359 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:20:36.359 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:20:36.359 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:20:36.360 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:20:36.360 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:20:36.360 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:20:36.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-26T00:20:36.672 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:20:36.672 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:20:36.672 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:20:37.051 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-03-26T00:20:37.051 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' 5 2024-03-26T00:20:37.051 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-26T00:20:37.051 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-26T00:20:37.052 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-26T00:20:37.052 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-26T00:20:37.052 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-26T00:20:37.052 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-26T00:20:37.052 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-26T00:20:37.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-26T00:20:37.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-26T00:20:37.407 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-26T00:20:37.407 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-26T00:20:37.407 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-26T00:20:37.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-26T00:20:37.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:20:37.609 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:20:37.609 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:20:37.609 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:20:37.609 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:20:37.610 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:20:37.610 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:20:37.921 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-26T00:20:37.922 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:20:37.922 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:20:37.922 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:20:38.302 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-03-26T00:20:38.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' 5 2024-03-26T00:20:38.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-26T00:20:38.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-26T00:20:38.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-26T00:20:38.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-26T00:20:38.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-26T00:20:38.303 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-26T00:20:38.304 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-26T00:20:38.683 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-26T00:20:38.683 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-26T00:20:38.683 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-26T00:20:38.683 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-26T00:20:38.683 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-26T00:20:39.084 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-26T00:20:39.084 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:20:39.085 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:20:39.085 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:20:39.085 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:20:39.085 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:20:39.085 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:20:39.085 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:20:39.385 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:20:39.385 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:20:39.386 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:20:39.386 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:20:39.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:20:39.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:20:39.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:20:39.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2106: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-03-26T00:20:39.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:20:39.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:20:39.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-26T00:20:39.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:20:39.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-03-26T00:20:39.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:20:39.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:20:39.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-26T00:20:39.754 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:20:39.754 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-26T00:20:39.754 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:20:39.754 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:20:39.754 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:20:39.754 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:20:39.754 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:20:41.063 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:20:41.063 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-03-26T00:20:41.063 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:20:41.063 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:20:41.063 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:20:41.064 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:20:41.064 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:20:41.064 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-03-26T00:20:41.733 INFO:tasks.workunit.client.0.smithi129.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-03-26T00:20:41.734 INFO:tasks.workunit.client.0.smithi129.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-03-26T00:20:42.275 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-26T00:20:42.275 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:20:42.275 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:20:42.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-26T00:20:42.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:20:42.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:20:42.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:20:42.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:20:42.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:20:42.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:20:42.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:20:42.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:20:42.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:20:42.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:20:42.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:20:42.276 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:20:42.276 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:20:42.277 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:20:42.277 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:20:42.277 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:20:42.277 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:20:42.277 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:20:42.277 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:20:42.277 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:20:42.277 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:20:42.277 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:20:42.277 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:20:42.277 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:20:42.277 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:20:42.277 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:20:42.277 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:20:42.277 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:20:42.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:20:42.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-26T00:20:42.278 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.1 2024-03-26T00:20:42.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:20:42.278 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-26T00:20:42.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-26T00:20:42.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:20:42.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:20:42.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:20:42.311 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:20:42.307+0000 7f78602bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:20:42.313 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:20:42.311+0000 7f78602bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:20:42.315 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:20:42.311+0000 7f78602bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:20:42.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-26T00:20:42.597 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:20:42.597 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:20:42.597 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:20:42.597 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:20:42.597 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:20:42.597 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:20:42.598 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:20:42.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:20:42.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:20:42.853 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:20:42.851+0000 7f78602bf7c0 -1 Falling back to public interface 2024-03-26T00:20:42.893 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:20:43.894 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:20:43.894 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:20:43.894 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:20:43.895 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:20:43.895 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:20:43.895 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:20:44.212 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:20:45.215 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:20:45.215 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:20:45.215 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:20:45.215 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:20:45.215 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:20:45.215 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:20:45.277 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:20:45.275+0000 7f78602bf7c0 -1 osd.1 203 log_to_monitors true 2024-03-26T00:20:45.530 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:20:46.531 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:20:46.531 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:20:46.531 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:20:46.531 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:20:46.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:20:46.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:20:46.964 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1 up in weight 1 up_from 207 up_thru 207 down_at 204 last_clean_interval [195,203) [v2:127.0.0.1:6810/2995450187,v1:127.0.0.1:6811/2995450187] [v2:127.0.0.1:6812/2995450187,v1:127.0.0.1:6813/2995450187] exists,up 0c2624cc-3bc6-4489-918a-a2155ff79e95 2024-03-26T00:20:46.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:20:46.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:20:46.965 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:20:46.965 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:20:46.965 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:20:46.965 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:20:46.965 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:20:46.965 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:20:46.966 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:20:46.967 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:20:46.967 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:20:46.967 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:20:46.967 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:20:47.089 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:20:47.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:20:47.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:20:47.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:20:47.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:20:47.090 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:20:47.405 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:20:47.405 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:20:47.405 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:20:47.405 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:20:47.406 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:20:47.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=863288426502 2024-03-26T00:20:47.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 863288426502 2024-03-26T00:20:47.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-863288426502' 2024-03-26T00:20:47.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:20:47.579 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:20:47.721 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=889058230274 2024-03-26T00:20:47.721 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 889058230274 2024-03-26T00:20:47.721 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-863288426502 1-889058230274' 2024-03-26T00:20:47.721 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:20:47.722 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-863288426502 2024-03-26T00:20:47.723 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:20:47.724 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:20:47.725 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-863288426502 2024-03-26T00:20:47.725 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:20:47.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=863288426502 2024-03-26T00:20:47.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 863288426502' 2024-03-26T00:20:47.727 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 863288426502 2024-03-26T00:20:47.727 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:20:48.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 863288426500 -lt 863288426502 2024-03-26T00:20:48.034 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:20:49.035 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:20:49.037 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:20:49.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 863288426502 -lt 863288426502 2024-03-26T00:20:49.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:20:49.350 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-889058230274 2024-03-26T00:20:49.350 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:20:49.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:20:49.354 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-889058230274 2024-03-26T00:20:49.354 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:20:49.355 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=889058230274 2024-03-26T00:20:49.356 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 889058230274' 2024-03-26T00:20:49.356 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 889058230274 2024-03-26T00:20:49.356 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:20:49.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 889058230274 -lt 889058230274 2024-03-26T00:20:49.690 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:20:49.690 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:20:49.690 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:20:50.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:20:50.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:20:50.046 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:20:50.046 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:20:50.046 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:20:50.046 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:20:50.047 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:20:50.047 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:20:50.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:20:50.391 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:20:50.391 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:20:50.392 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:20:50.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:20:50.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:20:50.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:20:50.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2107: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi 2024-03-26T00:20:50.754 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2109: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2024-03-26T00:20:50.754 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-03-26T00:20:50.754 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-26T00:20:50.754 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-03-26T00:20:50.754 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-26T00:20:50.754 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-03-26T00:20:50.754 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-26T00:20:50.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ11 2024-03-26T00:20:50.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-26T00:20:50.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-03-26T00:20:50.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-26T00:20:50.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-03-26T00:20:50.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-26T00:20:50.755 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ11 2024-03-26T00:20:50.755 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-03-26T00:20:50.755 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ11 2024-03-26T00:20:50.755 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ11 2024-03-26T00:20:50.755 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-26T00:20:51.068 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-03-26T00:20:51.068 INFO:tasks.workunit.client.0.smithi129.stderr:0' 2024-03-26T00:20:51.068 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-03-26T00:20:51.068 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-03-26T00:20:51.068 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-26T00:20:51.068 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-03-26T00:20:51.068 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-03-26T00:20:51.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-03-26T00:20:51.069 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-03-26T00:20:51.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-26T00:20:51.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-03-26T00:20:51.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-26T00:20:51.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-03-26T00:20:51.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-26T00:20:51.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-26T00:20:51.071 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-26T00:20:51.071 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-03-26T00:20:51.071 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-26T00:20:51.071 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-26T00:20:51.072 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-26T00:20:51.072 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:20:51.072 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:20:51.072 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-osd.1.asok 2024-03-26T00:20:51.072 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61636/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-03-26T00:20:51.180 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-26T00:20:51.181 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-26T00:20:51.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-26T00:20:51.182 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-03-26T00:20:51.182 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-26T00:20:51.182 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-26T00:20:51.183 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-26T00:20:51.183 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:20:51.183 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:20:51.183 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-osd.1.asok 2024-03-26T00:20:51.183 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-26T00:20:51.184 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.61636/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2024-03-26T00:20:51.291 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2110: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2024-03-26T00:20:51.292 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-03-26T00:20:51.292 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-26T00:20:51.292 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-03-26T00:20:51.292 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-26T00:20:51.292 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-03-26T00:20:51.292 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-26T00:20:51.292 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ12 2024-03-26T00:20:51.292 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-26T00:20:51.292 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-03-26T00:20:51.292 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-26T00:20:51.293 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-03-26T00:20:51.293 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-26T00:20:51.293 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ12 2024-03-26T00:20:51.293 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-03-26T00:20:51.293 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ12 2024-03-26T00:20:51.293 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ12 2024-03-26T00:20:51.293 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-26T00:20:51.584 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-03-26T00:20:51.584 INFO:tasks.workunit.client.0.smithi129.stderr:0' 2024-03-26T00:20:51.584 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-03-26T00:20:51.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-03-26T00:20:51.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-26T00:20:51.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-03-26T00:20:51.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-03-26T00:20:51.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-03-26T00:20:51.586 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-03-26T00:20:51.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-26T00:20:51.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-03-26T00:20:51.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-26T00:20:51.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-03-26T00:20:51.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-26T00:20:51.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-26T00:20:51.587 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-26T00:20:51.587 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-03-26T00:20:51.587 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-26T00:20:51.587 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-26T00:20:51.588 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-26T00:20:51.588 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:20:51.588 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:20:51.588 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-osd.0.asok 2024-03-26T00:20:51.588 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61636/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-03-26T00:20:51.696 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-26T00:20:51.696 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-26T00:20:51.699 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-26T00:20:51.699 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-03-26T00:20:51.699 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-26T00:20:51.700 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-26T00:20:51.700 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-26T00:20:51.700 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:20:51.700 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:20:51.700 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-osd.0.asok 2024-03-26T00:20:51.701 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-26T00:20:51.701 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.61636/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2024-03-26T00:20:51.816 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2111: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2024-03-26T00:20:51.816 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-03-26T00:20:51.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-26T00:20:51.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-03-26T00:20:51.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-26T00:20:51.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-03-26T00:20:51.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-26T00:20:51.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-03-26T00:20:51.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-26T00:20:51.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-03-26T00:20:51.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-26T00:20:51.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-03-26T00:20:51.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-26T00:20:51.817 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-03-26T00:20:51.817 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-03-26T00:20:51.817 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-03-26T00:20:51.818 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-03-26T00:20:51.818 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-26T00:20:52.122 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-03-26T00:20:52.123 INFO:tasks.workunit.client.0.smithi129.stderr:0' 2024-03-26T00:20:52.123 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-03-26T00:20:52.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-03-26T00:20:52.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-26T00:20:52.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-03-26T00:20:52.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-03-26T00:20:52.123 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-03-26T00:20:52.123 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-03-26T00:20:52.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-26T00:20:52.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-03-26T00:20:52.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-26T00:20:52.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-03-26T00:20:52.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-26T00:20:52.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-26T00:20:52.125 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-26T00:20:52.125 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-03-26T00:20:52.125 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-26T00:20:52.126 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-26T00:20:52.126 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-26T00:20:52.126 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:20:52.127 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:20:52.127 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-osd.0.asok 2024-03-26T00:20:52.127 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61636/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-03-26T00:20:52.233 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-26T00:20:52.233 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-26T00:20:52.234 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-26T00:20:52.235 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-03-26T00:20:52.235 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-26T00:20:52.235 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-26T00:20:52.236 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-26T00:20:52.236 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:20:52.236 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:20:52.236 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-osd.0.asok 2024-03-26T00:20:52.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-26T00:20:52.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.61636/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2024-03-26T00:20:52.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2112: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2024-03-26T00:20:52.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-03-26T00:20:52.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-26T00:20:52.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-03-26T00:20:52.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-26T00:20:52.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-03-26T00:20:52.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-26T00:20:52.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-03-26T00:20:52.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-26T00:20:52.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-03-26T00:20:52.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-26T00:20:52.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-03-26T00:20:52.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-26T00:20:52.349 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-03-26T00:20:52.349 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-03-26T00:20:52.349 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-03-26T00:20:52.350 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-03-26T00:20:52.350 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-26T00:20:52.659 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-03-26T00:20:52.660 INFO:tasks.workunit.client.0.smithi129.stderr:0' 2024-03-26T00:20:52.660 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-03-26T00:20:52.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-03-26T00:20:52.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-26T00:20:52.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-03-26T00:20:52.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-03-26T00:20:52.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-03-26T00:20:52.660 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-03-26T00:20:52.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-26T00:20:52.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-03-26T00:20:52.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-26T00:20:52.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-03-26T00:20:52.662 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-26T00:20:52.662 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-26T00:20:52.663 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-26T00:20:52.664 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-03-26T00:20:52.664 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-26T00:20:52.664 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-26T00:20:52.664 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-26T00:20:52.664 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:20:52.664 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:20:52.664 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-osd.1.asok 2024-03-26T00:20:52.665 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61636/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-03-26T00:20:52.771 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-26T00:20:52.772 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-26T00:20:52.773 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-26T00:20:52.773 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-03-26T00:20:52.773 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-26T00:20:52.773 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-26T00:20:52.773 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-26T00:20:52.773 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:20:52.773 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:20:52.774 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-osd.1.asok 2024-03-26T00:20:52.774 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-26T00:20:52.774 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.61636/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2024-03-26T00:20:52.883 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2115: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=134217728 2024-03-26T00:20:53.015 INFO:tasks.workunit.client.0.smithi129.stdout:osd.0: {} 2024-03-26T00:20:53.015 INFO:tasks.workunit.client.0.smithi129.stderr:osd.0: osd_max_object_size = '' 2024-03-26T00:20:53.031 INFO:tasks.workunit.client.0.smithi129.stdout:osd.1: {} 2024-03-26T00:20:53.031 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1: osd_max_object_size = '' 2024-03-26T00:20:53.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2117: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2024-03-26T00:20:53.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-03-26T00:20:53.046 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-26T00:20:53.046 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:20:53.046 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-26T00:20:53.047 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:20:53.047 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-26T00:20:53.357 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-03-26T00:15:16.083704+0000 2024-03-26T00:20:53.357 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-03-26T00:20:53.633 INFO:tasks.workunit.client.0.smithi129.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2024-03-26T00:20:53.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-03-26T00:15:16.083704+0000 last_deep_scrub_stamp 2024-03-26T00:20:53.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-26T00:20:53.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-26T00:15:16.083704+0000 2024-03-26T00:20:53.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-03-26T00:20:53.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-26T00:20:53.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:20:53.650 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-26T00:20:53.650 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:20:53.650 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-26T00:20:53.650 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-26T00:20:53.651 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:20:53.947 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:15:16.083704+0000 '>' 2024-03-26T00:15:16.083704+0000 2024-03-26T00:20:53.947 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:20:54.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:20:54.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:20:54.949 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-26T00:20:54.950 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:20:54.950 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-26T00:20:54.950 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:20:54.950 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-26T00:20:55.292 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:15:16.083704+0000 '>' 2024-03-26T00:15:16.083704+0000 2024-03-26T00:20:55.292 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:20:56.293 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:20:56.293 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:20:56.294 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-26T00:20:56.295 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:20:56.295 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-26T00:20:56.295 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:20:56.295 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-26T00:20:56.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:20:54.418349+0000 '>' 2024-03-26T00:15:16.083704+0000 2024-03-26T00:20:56.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-26T00:20:56.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2119: TEST_corrupt_scrub_replicated: err_strings=() 2024-03-26T00:20:56.616 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2120: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2024-03-26T00:20:56.616 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2121: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' 2024-03-26T00:20:56.616 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2122: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' 2024-03-26T00:20:56.616 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2123: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' 2024-03-26T00:20:56.616 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2124: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2024-03-26T00:20:56.616 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2125: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2024-03-26T00:20:56.616 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2126: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2024-03-26T00:20:56.616 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2127: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2024-03-26T00:20:56.616 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2128: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' 2024-03-26T00:20:56.616 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2129: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2024-03-26T00:20:56.617 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2130: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2024-03-26T00:20:56.617 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2131: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' 2024-03-26T00:20:56.617 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2132: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' 2024-03-26T00:20:56.617 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2133: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' 2024-03-26T00:20:56.617 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2134: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' 2024-03-26T00:20:56.617 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2135: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2024-03-26T00:20:56.618 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2136: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2024-03-26T00:20:56.618 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2137: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' 2024-03-26T00:20:56.618 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2138: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2024-03-26T00:20:56.618 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2139: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2024-03-26T00:20:56.618 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2140: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2024-03-26T00:20:56.618 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2141: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2024-03-26T00:20:56.618 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2142: TEST_corrupt_scrub_replicated: err_strings[22]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2024-03-26T00:20:56.619 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2143: TEST_corrupt_scrub_replicated: err_strings[23]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2024-03-26T00:20:56.619 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2144: TEST_corrupt_scrub_replicated: err_strings[24]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' 2024-03-26T00:20:56.619 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2145: TEST_corrupt_scrub_replicated: err_strings[25]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2024-03-26T00:20:56.619 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2146: TEST_corrupt_scrub_replicated: err_strings[26]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' 2024-03-26T00:20:56.619 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2147: TEST_corrupt_scrub_replicated: err_strings[27]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' 2024-03-26T00:20:56.619 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2148: TEST_corrupt_scrub_replicated: err_strings[28]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2024-03-26T00:20:56.619 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2149: TEST_corrupt_scrub_replicated: err_strings[29]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' 2024-03-26T00:20:56.619 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2150: TEST_corrupt_scrub_replicated: err_strings[30]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' 2024-03-26T00:20:56.620 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2151: TEST_corrupt_scrub_replicated: err_strings[31]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' 2024-03-26T00:20:56.620 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2152: TEST_corrupt_scrub_replicated: err_strings[32]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' 2024-03-26T00:20:56.620 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2153: TEST_corrupt_scrub_replicated: err_strings[33]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2024-03-26T00:20:56.621 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2154: TEST_corrupt_scrub_replicated: err_strings[34]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2024-03-26T00:20:56.621 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2155: TEST_corrupt_scrub_replicated: err_strings[35]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' 2024-03-26T00:20:56.621 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2156: TEST_corrupt_scrub_replicated: err_strings[36]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-03-26T00:20:56.621 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2157: TEST_corrupt_scrub_replicated: err_strings[37]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' 2024-03-26T00:20:56.621 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2158: TEST_corrupt_scrub_replicated: err_strings[38]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' 2024-03-26T00:20:56.621 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:20:56.621 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-03-26T00:20:56.624 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:20:56.624 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-03-26T00:20:56.636 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:20:56.636 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-26T00:20:56.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:20:56.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-03-26T00:20:56.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:20:56.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2024-03-26T00:20:56.672 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:20:56.673 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2024-03-26T00:20:56.683 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:20:56.683 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-03-26T00:20:56.694 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:20:56.695 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2024-03-26T00:20:56.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:20:56.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2024-03-26T00:20:56.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:20:56.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-26T00:20:56.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:20:56.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-26T00:20:56.739 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:20:56.739 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-03-26T00:20:56.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:20:56.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' td/osd-scrub-repair/osd.1.log 2024-03-26T00:20:56.764 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:20:56.764 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-26T00:20:56.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:20:56.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-26T00:20:56.788 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:20:56.788 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-26T00:20:56.799 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:20:56.799 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-26T00:20:56.811 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:20:56.811 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-03-26T00:20:56.823 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:20:56.823 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-03-26T00:20:56.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:20:56.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-03-26T00:20:56.844 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:20:56.844 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2024-03-26T00:20:56.855 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:20:56.856 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-03-26T00:20:56.867 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:20:56.867 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-03-26T00:20:56.877 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:20:56.877 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2024-03-26T00:20:56.890 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:20:56.891 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2024-03-26T00:20:56.903 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:20:56.903 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-03-26T00:20:56.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:20:56.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-03-26T00:20:56.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:20:56.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-26T00:20:56.937 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:20:56.937 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2024-03-26T00:20:56.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:20:56.950 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-26T00:20:56.961 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:20:56.961 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' td/osd-scrub-repair/osd.1.log 2024-03-26T00:20:56.974 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:20:56.974 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-26T00:20:56.986 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:20:56.986 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-03-26T00:20:56.999 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:20:56.999 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-03-26T00:20:57.012 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:20:57.013 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2024-03-26T00:20:57.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:20:57.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' td/osd-scrub-repair/osd.1.log 2024-03-26T00:20:57.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:20:57.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-03-26T00:20:57.051 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:20:57.051 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-03-26T00:20:57.065 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-26T00:20:57.065 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' td/osd-scrub-repair/osd.1.log 2024-03-26T00:20:57.080 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2169: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2024-03-26T00:20:57.115 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2171: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2024-03-26T00:20:57.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2171: TEST_corrupt_scrub_replicated: test 1 = 1 2024-03-26T00:20:57.146 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2024-03-26T00:20:57.174 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2024-03-26T00:20:57.174 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2024-03-26T00:20:57.203 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2024-03-26T00:20:57.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: epoch=207 2024-03-26T00:20:57.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-03-26T00:20:57.232 INFO:tasks.workunit.client.0.smithi129.stderr: . as $in 2024-03-26T00:20:57.232 INFO:tasks.workunit.client.0.smithi129.stderr: | if type == "object" then 2024-03-26T00:20:57.232 INFO:tasks.workunit.client.0.smithi129.stderr: reduce keys[] as $key 2024-03-26T00:20:57.232 INFO:tasks.workunit.client.0.smithi129.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-26T00:20:57.233 INFO:tasks.workunit.client.0.smithi129.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-26T00:20:57.233 INFO:tasks.workunit.client.0.smithi129.stderr: else f 2024-03-26T00:20:57.233 INFO:tasks.workunit.client.0.smithi129.stderr: end; 2024-03-26T00:20:57.233 INFO:tasks.workunit.client.0.smithi129.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-26T00:20:57.233 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-26T00:20:57.233 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-26T00:20:57.233 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-26T00:20:57.233 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-03-26T00:20:57.233 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-03-26T00:20:57.233 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-26T00:20:57.288 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-03-26T00:20:57.288 INFO:tasks.workunit.client.0.smithi129.stderr: . as $in 2024-03-26T00:20:57.288 INFO:tasks.workunit.client.0.smithi129.stderr: | if type == "object" then 2024-03-26T00:20:57.288 INFO:tasks.workunit.client.0.smithi129.stderr: reduce keys[] as $key 2024-03-26T00:20:57.288 INFO:tasks.workunit.client.0.smithi129.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-26T00:20:57.288 INFO:tasks.workunit.client.0.smithi129.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-26T00:20:57.288 INFO:tasks.workunit.client.0.smithi129.stderr: else f 2024-03-26T00:20:57.288 INFO:tasks.workunit.client.0.smithi129.stderr: end; 2024-03-26T00:20:57.289 INFO:tasks.workunit.client.0.smithi129.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-26T00:20:57.289 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-26T00:20:57.289 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-26T00:20:57.289 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-26T00:20:57.289 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-03-26T00:20:57.289 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-03-26T00:20:57.289 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-26T00:20:57.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3559: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-26T00:20:57.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-26T00:20:57.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3560: TEST_corrupt_scrub_replicated: test no = yes 2024-03-26T00:20:57.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3565: TEST_corrupt_scrub_replicated: test '' = yes 2024-03-26T00:20:57.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3570: TEST_corrupt_scrub_replicated: repair 3.0 2024-03-26T00:20:57.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=3.0 2024-03-26T00:20:57.344 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 3.0 2024-03-26T00:20:57.344 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:20:57.344 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:20:57.345 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:20:57.345 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-26T00:20:57.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-26T00:20:54.418349+0000 2024-03-26T00:20:57.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 3.0 2024-03-26T00:20:57.945 INFO:tasks.workunit.client.0.smithi129.stderr:instructing pg 3.0 on osd.1 to repair 2024-03-26T00:20:57.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 3.0 2024-03-26T00:20:54.418349+0000 2024-03-26T00:20:57.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-26T00:20:57.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-26T00:20:54.418349+0000 2024-03-26T00:20:57.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-26T00:20:57.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-26T00:20:57.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:20:57.963 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-26T00:20:57.963 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:20:57.963 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:20:57.963 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-26T00:20:57.964 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:20:58.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:20:54.418349+0000 '>' 2024-03-26T00:20:54.418349+0000 2024-03-26T00:20:58.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:20:59.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:20:59.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:20:59.283 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-26T00:20:59.283 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:20:59.283 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:20:59.284 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:20:59.284 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-26T00:20:59.635 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:20:54.418349+0000 '>' 2024-03-26T00:20:54.418349+0000 2024-03-26T00:20:59.635 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:21:00.637 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:21:00.637 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:21:00.638 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-26T00:21:00.639 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:21:00.639 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:21:00.639 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:21:00.639 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-26T00:21:00.957 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:20:59.199304+0000 '>' 2024-03-26T00:20:54.418349+0000 2024-03-26T00:21:00.957 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-26T00:21:00.957 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3571: TEST_corrupt_scrub_replicated: wait_for_clean 2024-03-26T00:21:00.957 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:21:00.958 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:21:00.958 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:21:00.958 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:21:00.958 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:21:00.958 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:21:00.959 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:21:00.959 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:21:00.959 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:21:01.073 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:21:01.074 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:21:01.074 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:21:01.074 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:21:01.074 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:21:01.074 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:21:01.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:21:01.390 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:21:01.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:21:01.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:21:01.391 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:21:01.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=863288426505 2024-03-26T00:21:01.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 863288426505 2024-03-26T00:21:01.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-863288426505' 2024-03-26T00:21:01.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:21:01.567 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:21:01.709 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=889058230278 2024-03-26T00:21:01.709 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 889058230278 2024-03-26T00:21:01.709 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-863288426505 1-889058230278' 2024-03-26T00:21:01.709 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:21:01.710 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-863288426505 2024-03-26T00:21:01.710 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:21:01.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:21:01.712 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-863288426505 2024-03-26T00:21:01.712 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:21:01.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=863288426505 2024-03-26T00:21:01.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 863288426505' 2024-03-26T00:21:01.713 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 863288426505 2024-03-26T00:21:01.714 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:21:02.011 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 863288426504 -lt 863288426505 2024-03-26T00:21:02.012 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:21:03.013 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:21:03.014 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:21:03.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 863288426506 -lt 863288426505 2024-03-26T00:21:03.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:21:03.343 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:21:03.343 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-889058230278 2024-03-26T00:21:03.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:21:03.345 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-889058230278 2024-03-26T00:21:03.345 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:21:03.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=889058230278 2024-03-26T00:21:03.348 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 889058230278 2024-03-26T00:21:03.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 889058230278' 2024-03-26T00:21:03.348 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:21:03.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 889058230278 -lt 889058230278 2024-03-26T00:21:03.652 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:21:03.653 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:21:03.653 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:21:04.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:21:04.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:21:04.047 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:21:04.047 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:21:04.047 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:21:04.047 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:21:04.048 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:21:04.048 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:21:04.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:21:04.380 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:21:04.380 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:21:04.380 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:21:04.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:21:04.745 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:21:04.745 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:21:04.745 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3574: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ17 td/osd-scrub-repair/robj17.out 2024-03-26T00:21:04.769 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3575: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ18 td/osd-scrub-repair/robj18.out 2024-03-26T00:21:04.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3577: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2024-03-26T00:21:04.815 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3578: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2024-03-26T00:21:04.816 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3579: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2024-03-26T00:21:04.816 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3580: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2024-03-26T00:21:04.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3582: TEST_corrupt_scrub_replicated: '[' 0 '!=' 0 ']' 2024-03-26T00:21:04.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3588: TEST_corrupt_scrub_replicated: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2024-03-26T00:21:05.243 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'csr_pool' removed 2024-03-26T00:21:05.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-26T00:21:05.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-26T00:21:05.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-26T00:21:05.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-26T00:21:05.266 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:21:05.266 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:21:05.266 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:21:05.266 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:21:05.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:21:05.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:21:05.580 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-26T00:21:05.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-26T00:21:05.582 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-26T00:21:05.583 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-26T00:21:05.583 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-26T00:21:05.584 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-26T00:21:05.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:21:05.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-26T00:21:05.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-26T00:21:05.587 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:21:05.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-26T00:21:05.590 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-26T00:21:05.590 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-26T00:21:05.638 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-26T00:21:05.638 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:21:05.638 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:21:05.638 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-26T00:21:05.640 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-26T00:21:05.640 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-26T00:21:05.640 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-26T00:21:05.640 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-26T00:21:05.640 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-26T00:21:05.640 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-26T00:21:05.640 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-26T00:21:05.640 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-26T00:21:05.640 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-26T00:21:05.640 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:21:05.640 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:21:05.641 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:21:05.641 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:21:05.641 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:21:05.643 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:21:05.643 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-26T00:21:05.644 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-26T00:21:05.645 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-26T00:21:05.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-26T00:21:05.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-26T00:21:05.647 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-26T00:21:05.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:21:05.648 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-26T00:21:05.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-26T00:21:05.649 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:21:05.650 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-26T00:21:05.652 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-26T00:21:05.652 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-26T00:21:05.653 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-26T00:21:05.653 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:21:05.653 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:21:05.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-26T00:21:05.655 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-26T00:21:05.655 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-26T00:21:05.655 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-26T00:21:05.657 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-26T00:21:05.657 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:21:05.657 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:21:05.657 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61636 2024-03-26T00:21:05.660 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-26T00:21:05.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-26T00:21:05.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-26T00:21:05.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-26T00:21:05.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_snapset_scrub_rep td/osd-scrub-repair 2024-03-26T00:21:05.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5931: TEST_corrupt_snapset_scrub_rep: local dir=td/osd-scrub-repair 2024-03-26T00:21:05.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5932: TEST_corrupt_snapset_scrub_rep: local poolname=csr_pool 2024-03-26T00:21:05.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5933: TEST_corrupt_snapset_scrub_rep: local total_objs=2 2024-03-26T00:21:05.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5935: TEST_corrupt_snapset_scrub_rep: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-03-26T00:21:05.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-26T00:21:05.662 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-26T00:21:05.662 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-26T00:21:05.662 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-26T00:21:05.662 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-26T00:21:05.662 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-03-26T00:21:06.153 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-26T00:21:06.153 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:21:06.153 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:21:06.154 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:21:06.154 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:21:06.154 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:21:06.154 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:21:06.155 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-03-26T00:21:06.364 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-26T00:21:06.366 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-26T00:21:06.366 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-26T00:21:06.366 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-26T00:21:06.366 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-26T00:21:06.367 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-26T00:21:06.367 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-26T00:21:06.367 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-26T00:21:06.367 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-26T00:21:06.368 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-26T00:21:06.368 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:21:06.368 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:21:06.368 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-26T00:21:06.369 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-26T00:21:06.369 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get fsid 2024-03-26T00:21:06.481 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-26T00:21:06.481 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-26T00:21:06.481 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-26T00:21:06.482 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-26T00:21:06.482 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-26T00:21:06.482 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-26T00:21:06.482 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-26T00:21:06.482 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-26T00:21:06.483 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-26T00:21:06.483 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:21:06.483 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:21:06.483 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-26T00:21:06.484 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-26T00:21:06.484 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get mon_host 2024-03-26T00:21:06.584 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5936: TEST_corrupt_snapset_scrub_rep: run_mgr td/osd-scrub-repair x 2024-03-26T00:21:06.584 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-26T00:21:06.584 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-26T00:21:06.584 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-26T00:21:06.584 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-26T00:21:06.584 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-26T00:21:06.584 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-26T00:21:07.475 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-26T00:21:07.475 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:21:07.475 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:21:07.475 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:21:07.475 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:21:07.476 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:21:07.476 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:21:07.477 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-26T00:21:07.478 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-26T00:21:07.514 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5937: TEST_corrupt_snapset_scrub_rep: run_osd td/osd-scrub-repair 0 2024-03-26T00:21:07.514 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-26T00:21:07.514 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-26T00:21:07.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-26T00:21:07.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-26T00:21:07.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:21:07.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:21:07.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:21:07.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:21:07.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:21:07.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-26T00:21:07.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-26T00:21:07.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-26T00:21:07.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-26T00:21:07.515 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:21:07.516 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-26T00:21:07.516 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:21:07.516 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:21:07.516 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:21:07.516 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:21:07.516 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:21:07.516 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:21:07.516 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:21:07.516 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:21:07.516 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-26T00:21:07.516 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-26T00:21:07.516 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:21:07.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:21:07.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:21:07.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:21:07.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:21:07.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:21:07.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-26T00:21:07.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-26T00:21:07.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-26T00:21:07.518 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-26T00:21:07.518 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=138922f9-13dd-4a89-83ce-237a7dedccb6 2024-03-26T00:21:07.518 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 138922f9-13dd-4a89-83ce-237a7dedccb6' 2024-03-26T00:21:07.518 INFO:tasks.workunit.client.0.smithi129.stdout:add osd0 138922f9-13dd-4a89-83ce-237a7dedccb6 2024-03-26T00:21:07.519 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-26T00:21:07.534 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDzFAJm0djnHxAAyrCOZFk69K1fqBqJow2mlQ== 2024-03-26T00:21:07.534 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDzFAJm0djnHxAAyrCOZFk69K1fqBqJow2mlQ=="}' 2024-03-26T00:21:07.534 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 138922f9-13dd-4a89-83ce-237a7dedccb6 -i td/osd-scrub-repair/0/new.json 2024-03-26T00:21:07.793 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-26T00:21:07.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-26T00:21:07.811 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDzFAJm0djnHxAAyrCOZFk69K1fqBqJow2mlQ== --osd-uuid 138922f9-13dd-4a89-83ce-237a7dedccb6 2024-03-26T00:21:07.841 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:21:07.838+0000 7fc0254397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:21:07.863 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:21:07.854+0000 7fc0254397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:21:07.863 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:21:07.858+0000 7fc0254397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:21:07.863 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:21:07.858+0000 7fc0254397c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-26T00:21:07.863 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:21:07.858+0000 7fc0254397c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-26T00:21:07.863 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:21:07.858+0000 7fc0254397c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-26T00:21:07.952 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:21:07.946+0000 7fc0254397c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-26T00:21:14.290 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-26T00:21:14.291 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-26T00:21:14.292 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd0 key to auth repository 2024-03-26T00:21:14.292 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-26T00:21:14.292 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-26T00:21:14.669 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.0 2024-03-26T00:21:14.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-26T00:21:14.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:21:14.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-26T00:21:14.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-26T00:21:14.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-26T00:21:14.700 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:21:14.698+0000 7fc8a008f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:21:14.702 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:21:14.698+0000 7fc8a008f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:21:14.704 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:21:14.702+0000 7fc8a008f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:21:15.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-26T00:21:15.022 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:21:15.022 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-26T00:21:15.022 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:21:15.022 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:21:15.022 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:21:15.022 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:21:15.022 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-26T00:21:15.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:21:15.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:21:15.238 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:21:15.234+0000 7fc8a008f7c0 -1 Falling back to public interface 2024-03-26T00:21:15.357 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:21:16.358 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:21:16.358 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:21:16.358 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:21:16.359 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-26T00:21:16.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:21:16.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:21:16.682 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:21:17.684 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:21:17.684 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:21:17.684 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-26T00:21:17.685 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:21:17.685 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:21:17.685 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:21:17.866 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:21:17.862+0000 7fc8a008f7c0 -1 osd.0 0 log_to_monitors true 2024-03-26T00:21:18.009 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:21:19.014 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:21:19.014 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:21:19.014 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-26T00:21:19.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:21:19.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:21:19.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:21:19.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:21:20.334 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:21:20.334 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:21:20.334 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-26T00:21:20.335 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-26T00:21:20.335 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:21:20.335 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:21:20.748 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:21:21.748 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:21:21.748 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:21:21.749 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-26T00:21:21.749 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-26T00:21:21.749 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:21:21.749 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:21:22.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:21:23.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:21:23.071 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:21:23.071 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-26T00:21:23.071 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-26T00:21:23.072 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:21:23.072 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:21:23.395 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:21:24.396 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:21:24.396 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:21:24.397 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-26T00:21:24.397 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-26T00:21:24.397 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:21:24.397 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:21:24.773 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:21:25.775 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:21:25.775 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:21:25.775 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-26T00:21:25.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-26T00:21:25.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:21:25.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:21:26.094 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:21:27.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:21:27.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:21:27.096 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-26T00:21:27.096 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-26T00:21:27.096 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:21:27.096 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:21:27.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:21:28.410 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:21:28.410 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:21:28.411 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-26T00:21:28.411 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-26T00:21:28.411 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:21:28.411 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:21:28.804 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:21:29.807 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:21:29.807 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:21:29.807 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-26T00:21:29.808 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-26T00:21:29.808 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:21:29.808 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:21:30.130 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:21:31.131 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:21:31.131 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:21:31.132 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-26T00:21:31.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-26T00:21:31.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:21:31.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:21:31.455 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:21:32.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:21:32.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:21:32.457 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-26T00:21:32.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-26T00:21:32.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:21:32.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:21:32.953 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:21:33.955 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:21:33.955 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:21:33.956 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-26T00:21:33.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-26T00:21:33.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:21:33.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:21:34.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:21:35.763 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:21:35.763 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:21:35.764 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-26T00:21:35.764 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-26T00:21:35.764 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:21:35.764 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:21:36.380 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:21:37.381 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:21:37.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:21:37.382 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-26T00:21:37.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-26T00:21:37.383 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:21:37.383 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:21:38.198 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:21:39.200 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:21:39.200 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:21:39.200 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-26T00:21:39.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-26T00:21:39.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:21:39.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:21:39.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:21:40.754 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:21:40.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:21:40.755 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-26T00:21:40.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-26T00:21:40.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:21:40.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:21:41.325 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:21:42.327 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:21:42.327 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:21:42.327 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-26T00:21:42.328 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-26T00:21:42.328 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:21:42.328 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:21:42.651 INFO:tasks.workunit.client.0.smithi129.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3166745085,v1:127.0.0.1:6803/3166745085] [v2:127.0.0.1:6804/3166745085,v1:127.0.0.1:6805/3166745085] exists,up 138922f9-13dd-4a89-83ce-237a7dedccb6 2024-03-26T00:21:42.658 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:21:42.658 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:21:42.658 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:21:42.658 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5938: TEST_corrupt_snapset_scrub_rep: run_osd td/osd-scrub-repair 1 2024-03-26T00:21:42.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-26T00:21:42.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-26T00:21:42.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-26T00:21:42.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-26T00:21:42.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:21:42.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:21:42.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:21:42.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:21:42.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:21:42.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:21:42.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:21:42.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-26T00:21:42.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-26T00:21:42.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:21:42.660 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-26T00:21:42.660 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:21:42.660 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:21:42.660 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:21:42.660 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:21:42.660 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:21:42.660 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:21:42.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:21:42.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:21:42.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-26T00:21:42.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-26T00:21:42.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:21:42.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:21:42.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:21:42.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:21:42.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:21:42.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:21:42.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-26T00:21:42.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-26T00:21:42.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:21:42.661 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-26T00:21:42.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dcc051ca-171d-41f6-9416-056b15a50ae2 2024-03-26T00:21:42.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 dcc051ca-171d-41f6-9416-056b15a50ae2' 2024-03-26T00:21:42.662 INFO:tasks.workunit.client.0.smithi129.stdout:add osd1 dcc051ca-171d-41f6-9416-056b15a50ae2 2024-03-26T00:21:42.662 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-26T00:21:42.681 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAWFQJmoz+gKBAA6lLvKJdUyLMw3ZIu/SJBpA== 2024-03-26T00:21:42.681 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAWFQJmoz+gKBAA6lLvKJdUyLMw3ZIu/SJBpA=="}' 2024-03-26T00:21:42.681 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dcc051ca-171d-41f6-9416-056b15a50ae2 -i td/osd-scrub-repair/1/new.json 2024-03-26T00:21:43.038 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-26T00:21:43.056 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-26T00:21:43.058 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAWFQJmoz+gKBAA6lLvKJdUyLMw3ZIu/SJBpA== --osd-uuid dcc051ca-171d-41f6-9416-056b15a50ae2 2024-03-26T00:21:43.096 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:21:43.098+0000 7f8d403eb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:21:43.122 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:21:43.122+0000 7f8d403eb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:21:43.124 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:21:43.122+0000 7f8d403eb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:21:43.124 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:21:43.126+0000 7f8d403eb7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-26T00:21:43.124 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:21:43.126+0000 7f8d403eb7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-26T00:21:43.124 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:21:43.126+0000 7f8d403eb7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-26T00:21:43.194 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:21:43.194+0000 7f8d403eb7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-26T00:21:50.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-26T00:21:50.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-26T00:21:50.103 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd1 key to auth repository 2024-03-26T00:21:50.104 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-26T00:21:50.104 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-26T00:21:50.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-26T00:21:50.498 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.1 2024-03-26T00:21:50.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:21:50.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-26T00:21:50.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-26T00:21:50.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-26T00:21:50.529 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:21:50.529+0000 7fb1deb607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:21:50.561 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:21:50.561+0000 7fb1deb607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:21:50.562 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:21:50.561+0000 7fb1deb607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:21:50.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-26T00:21:50.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:21:50.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:21:50.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:21:50.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:21:50.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:21:50.834 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-26T00:21:50.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:21:50.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:21:50.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:21:51.115 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:21:51.113+0000 7fb1deb607c0 -1 Falling back to public interface 2024-03-26T00:21:51.163 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:21:52.163 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:21:52.164 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:21:52.164 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-26T00:21:52.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:21:52.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:21:52.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:21:52.480 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:21:53.482 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:21:53.482 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:21:53.482 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-26T00:21:53.483 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:21:53.483 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:21:53.483 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:21:53.675 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:21:53.673+0000 7fb1deb607c0 -1 osd.1 0 log_to_monitors true 2024-03-26T00:21:53.807 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:21:54.809 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:21:54.809 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:21:54.810 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-26T00:21:54.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:21:54.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:21:54.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:21:55.137 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:21:56.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:21:56.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:21:56.139 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-26T00:21:56.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-26T00:21:56.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:21:56.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:21:56.441 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:21:57.443 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:21:57.444 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:21:57.444 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-26T00:21:57.444 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-26T00:21:57.444 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:21:57.445 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:21:57.765 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:21:58.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:21:58.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:21:58.768 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-26T00:21:58.768 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-26T00:21:58.768 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:21:58.768 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:21:59.181 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:22:00.181 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:22:00.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:22:00.182 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-26T00:22:00.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-26T00:22:00.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:22:00.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:22:00.502 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:22:01.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:22:01.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:22:01.505 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-26T00:22:01.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-26T00:22:01.506 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:22:01.506 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:22:01.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:22:02.822 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:22:02.823 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:22:02.823 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-26T00:22:02.823 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-26T00:22:02.824 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:22:02.824 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:22:03.212 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:22:04.213 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:22:04.213 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:22:04.214 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-26T00:22:04.214 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-26T00:22:04.214 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:22:04.214 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:22:04.602 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:22:05.604 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:22:05.604 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:22:05.604 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-26T00:22:05.605 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-26T00:22:05.605 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:22:05.605 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:22:05.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:22:06.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:22:06.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:22:06.924 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-26T00:22:06.924 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-26T00:22:06.924 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:22:06.924 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:22:07.253 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:22:08.255 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:22:08.255 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:22:08.255 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-26T00:22:08.255 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-26T00:22:08.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:22:08.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:22:08.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:22:09.628 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:22:09.628 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:22:09.629 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-26T00:22:09.629 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-26T00:22:09.629 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:22:09.629 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:22:10.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:22:11.763 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:22:11.763 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:22:11.763 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-26T00:22:11.763 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-26T00:22:11.764 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:22:11.764 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:22:12.117 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:22:13.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:22:13.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:22:13.119 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-26T00:22:13.120 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-26T00:22:13.120 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:22:13.120 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:22:13.675 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:22:14.677 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:22:14.677 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:22:14.678 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-26T00:22:14.678 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-26T00:22:14.678 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:22:14.678 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:22:15.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:22:16.463 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:22:16.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:22:16.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-26T00:22:16.464 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-26T00:22:16.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:22:16.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:22:17.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:22:18.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:22:18.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:22:18.024 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-26T00:22:18.024 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-26T00:22:18.024 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:22:18.024 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:22:18.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:22:19.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:22:19.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:22:19.504 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-26T00:22:19.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-26T00:22:19.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:22:19.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:22:19.818 INFO:tasks.workunit.client.0.smithi129.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1476130068,v1:127.0.0.1:6811/1476130068] [v2:127.0.0.1:6812/1476130068,v1:127.0.0.1:6813/1476130068] exists,up dcc051ca-171d-41f6-9416-056b15a50ae2 2024-03-26T00:22:19.819 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:22:19.819 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:22:19.819 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:22:19.819 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5939: TEST_corrupt_snapset_scrub_rep: create_rbd_pool 2024-03-26T00:22:19.819 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-26T00:22:20.119 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'rbd' does not exist 2024-03-26T00:22:20.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-26T00:22:20.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-26T00:22:20.545 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'rbd' created 2024-03-26T00:22:20.565 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-26T00:22:21.570 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-26T00:22:23.540 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5940: TEST_corrupt_snapset_scrub_rep: wait_for_clean 2024-03-26T00:22:23.540 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:22:23.540 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:22:23.540 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:22:23.541 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:22:23.542 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:22:23.542 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:22:23.543 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:22:23.543 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:22:23.543 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:22:23.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:22:23.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:22:23.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:22:23.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:22:23.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:22:23.677 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:22:24.004 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:22:24.004 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:22:24.004 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:22:24.005 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:22:24.005 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:22:24.206 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836493 2024-03-26T00:22:24.206 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836493 2024-03-26T00:22:24.206 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493' 2024-03-26T00:22:24.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:22:24.207 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:22:24.355 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=38654705667 2024-03-26T00:22:24.355 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 38654705667 2024-03-26T00:22:24.355 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493 1-38654705667' 2024-03-26T00:22:24.355 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:22:24.355 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836493 2024-03-26T00:22:24.355 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:22:24.357 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:22:24.358 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836493 2024-03-26T00:22:24.358 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:22:24.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836493 2024-03-26T00:22:24.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2024-03-26T00:22:24.359 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 21474836493 2024-03-26T00:22:24.360 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:22:24.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836493 2024-03-26T00:22:24.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:22:25.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:22:25.690 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:22:26.013 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836493 2024-03-26T00:22:26.013 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:22:27.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-26T00:22:27.016 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:22:27.339 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836493 -lt 21474836493 2024-03-26T00:22:27.340 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:22:27.341 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-38654705667 2024-03-26T00:22:27.341 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:22:27.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:22:27.344 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-38654705667 2024-03-26T00:22:27.344 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:22:27.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=38654705667 2024-03-26T00:22:27.346 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 38654705667 2024-03-26T00:22:27.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 38654705667' 2024-03-26T00:22:27.347 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:22:27.718 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 38654705668 -lt 38654705667 2024-03-26T00:22:27.718 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:22:27.719 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:22:27.719 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:22:28.129 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-26T00:22:28.129 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:22:28.130 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:22:28.130 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:22:28.130 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:22:28.130 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:22:28.130 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:22:28.131 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:22:28.446 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-26T00:22:28.446 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:22:28.447 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:22:28.447 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:22:28.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-26T00:22:28.818 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:22:28.818 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:22:28.818 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5942: TEST_corrupt_snapset_scrub_rep: create_pool foo 1 2024-03-26T00:22:28.818 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-03-26T00:22:29.235 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'foo' created 2024-03-26T00:22:29.252 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-26T00:22:30.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5943: TEST_corrupt_snapset_scrub_rep: create_pool csr_pool 1 1 2024-03-26T00:22:30.254 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create csr_pool 1 1 2024-03-26T00:22:30.758 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'csr_pool' created 2024-03-26T00:22:30.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-26T00:22:31.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5944: TEST_corrupt_snapset_scrub_rep: wait_for_clean 2024-03-26T00:22:31.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:22:31.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:22:31.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:22:31.782 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:22:31.783 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:22:31.783 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:22:31.784 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:22:31.784 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:22:31.784 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:22:31.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:22:31.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:22:31.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:22:31.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:22:31.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:22:31.919 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:22:32.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:22:32.326 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:22:32.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:22:32.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:22:32.327 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:22:32.479 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836495 2024-03-26T00:22:32.479 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836495 2024-03-26T00:22:32.479 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836495' 2024-03-26T00:22:32.479 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:22:32.480 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:22:32.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=38654705670 2024-03-26T00:22:32.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 38654705670 2024-03-26T00:22:32.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836495 1-38654705670' 2024-03-26T00:22:32.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:22:32.661 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836495 2024-03-26T00:22:32.661 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:22:32.663 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:22:32.664 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836495 2024-03-26T00:22:32.665 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:22:32.666 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836495 2024-03-26T00:22:32.666 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 21474836495 2024-03-26T00:22:32.666 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2024-03-26T00:22:32.666 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:22:32.997 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836494 -lt 21474836495 2024-03-26T00:22:32.997 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:22:33.998 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:22:34.000 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:22:34.315 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836495 2024-03-26T00:22:34.315 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:22:34.316 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-38654705670 2024-03-26T00:22:34.317 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:22:34.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:22:34.320 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-38654705670 2024-03-26T00:22:34.320 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:22:34.321 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=38654705670 2024-03-26T00:22:34.321 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 38654705670 2024-03-26T00:22:34.321 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 38654705670' 2024-03-26T00:22:34.322 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:22:34.685 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 38654705670 -lt 38654705670 2024-03-26T00:22:34.685 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:22:34.686 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:22:34.686 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:22:35.057 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:22:35.057 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:22:35.057 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:22:35.058 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:22:35.058 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:22:35.058 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:22:35.058 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:22:35.058 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:22:35.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:22:35.391 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:22:35.391 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:22:35.391 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:22:35.745 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:22:35.745 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:22:35.745 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:22:35.746 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5946: TEST_corrupt_snapset_scrub_rep: seq 1 2 2024-03-26T00:22:35.747 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5946: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-03-26T00:22:35.747 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5947: TEST_corrupt_snapset_scrub_rep: objname=ROBJ1 2024-03-26T00:22:35.747 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5948: TEST_corrupt_snapset_scrub_rep: add_something td/osd-scrub-repair csr_pool ROBJ1 2024-03-26T00:22:35.748 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-26T00:22:35.748 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-26T00:22:35.748 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ1 2024-03-26T00:22:35.748 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-26T00:22:35.748 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-26T00:22:35.748 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-26T00:22:36.135 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-26T00:22:36.154 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-26T00:22:36.747 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-26T00:22:36.768 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-26T00:22:36.768 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-26T00:22:36.768 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:22:37.044 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5950: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2024-03-26T00:22:37.185 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5951: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2024-03-26T00:22:37.328 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5946: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-03-26T00:22:37.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5947: TEST_corrupt_snapset_scrub_rep: objname=ROBJ2 2024-03-26T00:22:37.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5948: TEST_corrupt_snapset_scrub_rep: add_something td/osd-scrub-repair csr_pool ROBJ2 2024-03-26T00:22:37.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-26T00:22:37.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-26T00:22:37.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ2 2024-03-26T00:22:37.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-26T00:22:37.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-26T00:22:37.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-26T00:22:37.741 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-26T00:22:37.762 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-26T00:22:38.260 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-26T00:22:38.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-26T00:22:38.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-26T00:22:38.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:22:38.486 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5950: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2024-03-26T00:22:38.726 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5951: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2024-03-26T00:22:38.859 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5954: TEST_corrupt_snapset_scrub_rep: get_pg csr_pool ROBJ0 2024-03-26T00:22:38.860 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=csr_pool 2024-03-26T00:22:38.860 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=ROBJ0 2024-03-26T00:22:38.860 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-03-26T00:22:38.860 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-26T00:22:39.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5954: TEST_corrupt_snapset_scrub_rep: local pg=3.0 2024-03-26T00:22:39.183 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5955: TEST_corrupt_snapset_scrub_rep: get_primary csr_pool ROBJ0 2024-03-26T00:22:39.183 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=csr_pool 2024-03-26T00:22:39.183 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=ROBJ0 2024-03-26T00:22:39.183 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map csr_pool ROBJ0 2024-03-26T00:22:39.183 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-26T00:22:39.535 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5955: TEST_corrupt_snapset_scrub_rep: local primary=1 2024-03-26T00:22:39.535 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5957: TEST_corrupt_snapset_scrub_rep: rados -p csr_pool mksnap snap1 2024-03-26T00:22:39.784 INFO:tasks.workunit.client.0.smithi129.stdout:created pool csr_pool snap snap1 2024-03-26T00:22:39.788 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5958: TEST_corrupt_snapset_scrub_rep: echo -n head_of_snapshot_data 2024-03-26T00:22:39.789 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: seq 1 2 2024-03-26T00:22:39.790 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-03-26T00:22:39.790 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5961: TEST_corrupt_snapset_scrub_rep: objname=ROBJ1 2024-03-26T00:22:39.791 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: expr 1 % 2 2024-03-26T00:22:39.793 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: local osd=1 2024-03-26T00:22:39.793 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5966: TEST_corrupt_snapset_scrub_rep: case $i in 2024-03-26T00:22:39.793 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5968: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/change 2024-03-26T00:22:40.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5969: TEST_corrupt_snapset_scrub_rep: objectstore_tool td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-03-26T00:22:40.063 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:22:40.063 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:22:40.063 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-26T00:22:40.063 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:22:40.063 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-03-26T00:22:40.063 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:22:40.063 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:22:40.063 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-26T00:22:40.063 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:22:40.063 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-26T00:22:40.064 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:22:40.064 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:22:40.064 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:22:40.064 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:22:40.064 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:22:40.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:22:40.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-03-26T00:22:40.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:22:40.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:22:40.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:22:40.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:22:40.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:22:40.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 --head ROBJ1 clear-snapset corrupt 2024-03-26T00:22:42.663 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-26T00:22:42.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:22:42.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:22:42.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-26T00:22:42.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:22:42.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:22:42.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:22:42.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:22:42.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:22:42.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:22:42.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:22:42.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:22:42.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:22:42.666 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:22:42.666 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:22:42.666 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:22:42.666 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:22:42.666 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:22:42.666 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:22:42.666 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:22:42.667 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:22:42.667 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:22:42.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:22:42.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:22:42.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:22:42.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:22:42.668 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:22:42.668 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:22:42.668 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:22:42.668 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:22:42.668 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:22:42.668 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:22:42.668 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:22:42.668 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-26T00:22:42.668 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.1 2024-03-26T00:22:42.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:22:42.670 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-26T00:22:42.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-26T00:22:42.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:22:42.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:22:42.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:22:42.697 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:22:42.696+0000 7f8bd0b9a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:22:42.699 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:22:42.696+0000 7f8bd0b9a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:22:42.700 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:22:42.700+0000 7f8bd0b9a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:22:42.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-26T00:22:42.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:22:42.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:22:42.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:22:42.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:22:42.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:22:42.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:22:42.988 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:22:42.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:22:42.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:22:43.227 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:22:43.224+0000 7f8bd0b9a7c0 -1 Falling back to public interface 2024-03-26T00:22:43.315 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:22:44.316 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:22:44.316 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:22:44.316 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:22:44.317 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:22:44.317 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:22:44.317 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:22:44.635 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:22:45.637 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:22:45.637 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:22:45.638 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:22:45.638 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:22:45.638 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:22:45.638 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:22:45.806 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:22:45.804+0000 7f8bd0b9a7c0 -1 osd.1 29 log_to_monitors true 2024-03-26T00:22:46.044 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:22:47.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:22:47.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:22:47.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:22:47.046 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:22:47.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:22:47.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:22:47.522 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1 up in weight 1 up_from 33 up_thru 33 down_at 30 last_clean_interval [9,29) [v2:127.0.0.1:6810/1194859092,v1:127.0.0.1:6811/1194859092] [v2:127.0.0.1:6812/1194859092,v1:127.0.0.1:6813/1194859092] exists,up dcc051ca-171d-41f6-9416-056b15a50ae2 2024-03-26T00:22:47.522 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:22:47.522 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:22:47.522 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:22:47.522 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:22:47.522 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:22:47.522 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:22:47.522 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:22:47.523 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:22:47.523 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:22:47.524 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:22:47.524 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:22:47.524 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:22:47.524 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:22:47.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:22:47.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:22:47.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:22:47.647 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:22:47.647 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:22:47.647 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:22:48.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:22:48.047 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:22:48.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:22:48.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:22:48.048 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:22:48.245 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836499 2024-03-26T00:22:48.245 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836499 2024-03-26T00:22:48.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499' 2024-03-26T00:22:48.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:22:48.246 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:22:48.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920770 2024-03-26T00:22:48.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920770 2024-03-26T00:22:48.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-141733920770' 2024-03-26T00:22:48.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:22:48.390 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836499 2024-03-26T00:22:48.390 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:22:48.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:22:48.392 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836499 2024-03-26T00:22:48.392 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:22:48.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836499 2024-03-26T00:22:48.395 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-03-26T00:22:48.395 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 21474836499 2024-03-26T00:22:48.395 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:22:48.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836499 2024-03-26T00:22:48.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:22:49.705 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:22:49.706 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:22:50.022 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836499 2024-03-26T00:22:50.022 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:22:51.024 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-26T00:22:51.025 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:22:51.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836499 2024-03-26T00:22:51.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:22:51.332 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-141733920770 2024-03-26T00:22:51.332 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:22:51.334 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:22:51.335 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-141733920770 2024-03-26T00:22:51.335 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:22:51.336 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920770 2024-03-26T00:22:51.336 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 141733920770' 2024-03-26T00:22:51.336 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 141733920770 2024-03-26T00:22:51.337 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:22:51.681 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920770 -lt 141733920770 2024-03-26T00:22:51.682 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:22:51.682 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:22:51.682 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:22:52.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:22:52.085 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:22:52.085 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:22:52.085 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:22:52.085 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:22:52.085 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:22:52.085 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:22:52.086 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:22:52.372 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:22:52.372 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:22:52.373 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:22:52.373 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:22:52.747 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:22:52.747 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:22:52.747 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:22:52.747 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-03-26T00:22:52.747 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5961: TEST_corrupt_snapset_scrub_rep: objname=ROBJ2 2024-03-26T00:22:52.747 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: expr 2 % 2 2024-03-26T00:22:52.749 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: local osd=0 2024-03-26T00:22:52.749 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5966: TEST_corrupt_snapset_scrub_rep: case $i in 2024-03-26T00:22:52.749 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5973: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/change 2024-03-26T00:22:52.901 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5974: TEST_corrupt_snapset_scrub_rep: objectstore_tool td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-03-26T00:22:52.901 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:22:52.902 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:22:52.902 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-26T00:22:52.902 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:22:52.902 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-03-26T00:22:52.902 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:22:52.902 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:22:52.902 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-26T00:22:52.902 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:22:52.902 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-26T00:22:52.903 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:22:52.903 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:22:52.903 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:22:52.903 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:22:52.903 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:22:53.214 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:22:53.214 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-03-26T00:22:53.214 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:22:53.214 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:22:53.214 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:22:53.214 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:22:53.215 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:22:53.215 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 --head ROBJ2 clear-snapset corrupt 2024-03-26T00:22:55.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-26T00:22:55.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:22:55.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:22:55.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-26T00:22:55.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:22:55.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:22:55.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:22:55.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:22:55.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:22:55.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:22:55.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-26T00:22:55.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-26T00:22:55.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:22:55.462 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:22:55.462 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:22:55.462 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:22:55.462 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:22:55.462 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:22:55.464 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:22:55.464 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:22:55.464 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:22:55.465 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:22:55.465 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:22:55.465 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:22:55.465 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:22:55.465 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:22:55.465 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:22:55.465 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:22:55.466 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:22:55.466 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:22:55.466 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:22:55.466 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:22:55.466 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-26T00:22:55.466 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-26T00:22:55.466 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.0 2024-03-26T00:22:55.468 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:22:55.468 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-26T00:22:55.468 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-26T00:22:55.469 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:22:55.469 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:22:55.469 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:22:55.499 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:22:55.496+0000 7fe01278a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:22:55.502 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:22:55.500+0000 7fe01278a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:22:55.503 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:22:55.500+0000 7fe01278a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:22:55.794 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-26T00:22:55.794 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:22:55.794 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-26T00:22:55.794 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:22:55.794 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:22:55.794 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:22:55.795 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:22:55.795 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:22:55.795 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:22:55.795 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:22:56.047 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:22:56.044+0000 7fe01278a7c0 -1 Falling back to public interface 2024-03-26T00:22:56.133 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:22:57.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:22:57.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:22:57.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:22:57.135 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:22:57.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:22:57.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:22:57.447 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:22:58.449 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:22:58.449 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:22:58.449 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:22:58.449 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:22:58.450 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:22:58.450 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:22:58.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:22:58.996 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:22:58.992+0000 7fe01278a7c0 -1 osd.0 35 log_to_monitors true 2024-03-26T00:22:59.812 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:22:59.812 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:22:59.812 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:22:59.813 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:22:59.813 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:22:59.813 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:23:00.133 INFO:tasks.workunit.client.0.smithi129.stderr:osd.0 up in weight 1 up_from 39 up_thru 39 down_at 36 last_clean_interval [5,35) [v2:127.0.0.1:6802/1880721174,v1:127.0.0.1:6803/1880721174] [v2:127.0.0.1:6804/1880721174,v1:127.0.0.1:6805/1880721174] exists,up 138922f9-13dd-4a89-83ce-237a7dedccb6 2024-03-26T00:23:00.134 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:23:00.134 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:23:00.134 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:23:00.134 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:23:00.134 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:23:00.134 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:23:00.134 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:23:00.134 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:23:00.135 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:23:00.135 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:23:00.135 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:23:00.135 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:23:00.135 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:23:00.242 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:23:00.242 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:23:00.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:23:00.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:23:00.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:23:00.243 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:23:00.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:23:00.599 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:23:00.600 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:23:00.600 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:23:00.600 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:23:00.744 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724546 2024-03-26T00:23:00.744 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724546 2024-03-26T00:23:00.744 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-167503724546' 2024-03-26T00:23:00.744 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:23:00.745 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:23:00.886 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920773 2024-03-26T00:23:00.886 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920773 2024-03-26T00:23:00.886 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-167503724546 1-141733920773' 2024-03-26T00:23:00.886 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:23:00.887 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-167503724546 2024-03-26T00:23:00.887 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:23:00.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:23:00.888 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-167503724546 2024-03-26T00:23:00.888 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:23:00.889 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724546 2024-03-26T00:23:00.889 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 167503724546' 2024-03-26T00:23:00.889 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 167503724546 2024-03-26T00:23:00.890 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:23:01.187 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 167503724546 2024-03-26T00:23:01.187 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:23:02.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:23:02.190 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:23:02.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724546 -lt 167503724546 2024-03-26T00:23:02.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:23:02.497 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-141733920773 2024-03-26T00:23:02.497 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:23:02.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:23:02.501 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-141733920773 2024-03-26T00:23:02.501 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:23:02.502 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920773 2024-03-26T00:23:02.502 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 141733920773' 2024-03-26T00:23:02.502 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 141733920773 2024-03-26T00:23:02.502 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:23:02.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920774 -lt 141733920773 2024-03-26T00:23:02.815 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:23:02.816 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:23:02.816 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:23:03.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-26T00:23:03.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:23:03.202 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:23:03.202 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:23:03.202 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:23:03.202 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:23:03.202 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:23:03.202 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:23:03.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-26T00:23:03.527 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:23:03.528 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:23:03.528 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:23:03.871 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-26T00:23:03.871 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:23:03.871 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:23:03.871 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5979: TEST_corrupt_snapset_scrub_rep: rm td/osd-scrub-repair/change 2024-03-26T00:23:03.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5981: TEST_corrupt_snapset_scrub_rep: pg_scrub 3.0 2024-03-26T00:23:03.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-03-26T00:23:03.872 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-03-26T00:23:03.873 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:23:03.873 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:23:03.873 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:23:03.873 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-26T00:23:04.215 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-03-26T00:22:30.633905+0000 2024-03-26T00:23:04.215 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-03-26T00:23:04.489 INFO:tasks.workunit.client.0.smithi129.stderr:instructing pg 3.0 on osd.1 to scrub 2024-03-26T00:23:04.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-03-26T00:22:30.633905+0000 2024-03-26T00:23:04.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-26T00:23:04.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-26T00:22:30.633905+0000 2024-03-26T00:23:04.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-26T00:23:04.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-26T00:23:04.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:23:04.506 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-26T00:23:04.506 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:23:04.506 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:23:04.506 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-26T00:23:04.506 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:23:04.815 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:22:30.633905+0000 '>' 2024-03-26T00:22:30.633905+0000 2024-03-26T00:23:04.816 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:23:05.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:23:05.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:23:05.818 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-26T00:23:05.818 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:23:05.818 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:23:05.819 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:23:05.819 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-26T00:23:06.156 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:22:30.633905+0000 '>' 2024-03-26T00:22:30.633905+0000 2024-03-26T00:23:06.156 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:23:07.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:23:07.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:23:07.158 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-26T00:23:07.159 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:23:07.159 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:23:07.159 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:23:07.159 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-26T00:23:07.462 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:22:30.633905+0000 '>' 2024-03-26T00:22:30.633905+0000 2024-03-26T00:23:07.462 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:23:08.463 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:23:08.463 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:23:08.464 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-26T00:23:08.464 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-26T00:23:08.464 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:23:08.465 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:23:08.465 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-26T00:23:08.811 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:23:05.471655+0000 '>' 2024-03-26T00:22:30.633905+0000 2024-03-26T00:23:08.811 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-26T00:23:08.811 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5983: TEST_corrupt_snapset_scrub_rep: rados list-inconsistent-pg csr_pool 2024-03-26T00:23:08.844 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5985: TEST_corrupt_snapset_scrub_rep: jq '. | length' td/osd-scrub-repair/json 2024-03-26T00:23:08.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5985: TEST_corrupt_snapset_scrub_rep: test 1 = 1 2024-03-26T00:23:08.872 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5987: TEST_corrupt_snapset_scrub_rep: jq -r '.[0]' td/osd-scrub-repair/json 2024-03-26T00:23:08.901 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5987: TEST_corrupt_snapset_scrub_rep: test 3.0 = 3.0 2024-03-26T00:23:08.901 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5989: TEST_corrupt_snapset_scrub_rep: rados list-inconsistent-obj 3.0 2024-03-26T00:23:08.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5991: TEST_corrupt_snapset_scrub_rep: jq 'def walk(f): 2024-03-26T00:23:08.930 INFO:tasks.workunit.client.0.smithi129.stderr: . as $in 2024-03-26T00:23:08.930 INFO:tasks.workunit.client.0.smithi129.stderr: | if type == "object" then 2024-03-26T00:23:08.930 INFO:tasks.workunit.client.0.smithi129.stderr: reduce keys[] as $key 2024-03-26T00:23:08.930 INFO:tasks.workunit.client.0.smithi129.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-26T00:23:08.930 INFO:tasks.workunit.client.0.smithi129.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-26T00:23:08.930 INFO:tasks.workunit.client.0.smithi129.stderr: else f 2024-03-26T00:23:08.930 INFO:tasks.workunit.client.0.smithi129.stderr: end; 2024-03-26T00:23:08.930 INFO:tasks.workunit.client.0.smithi129.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-26T00:23:08.930 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-26T00:23:08.930 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-26T00:23:08.930 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-26T00:23:08.930 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-03-26T00:23:08.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5991: TEST_corrupt_snapset_scrub_rep: jq .inconsistents 2024-03-26T00:23:08.931 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5991: TEST_corrupt_snapset_scrub_rep: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-26T00:23:08.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6157: TEST_corrupt_snapset_scrub_rep: jq 'def walk(f): 2024-03-26T00:23:08.966 INFO:tasks.workunit.client.0.smithi129.stderr: . as $in 2024-03-26T00:23:08.966 INFO:tasks.workunit.client.0.smithi129.stderr: | if type == "object" then 2024-03-26T00:23:08.966 INFO:tasks.workunit.client.0.smithi129.stderr: reduce keys[] as $key 2024-03-26T00:23:08.966 INFO:tasks.workunit.client.0.smithi129.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-26T00:23:08.966 INFO:tasks.workunit.client.0.smithi129.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-26T00:23:08.967 INFO:tasks.workunit.client.0.smithi129.stderr: else f 2024-03-26T00:23:08.967 INFO:tasks.workunit.client.0.smithi129.stderr: end; 2024-03-26T00:23:08.967 INFO:tasks.workunit.client.0.smithi129.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-26T00:23:08.967 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-26T00:23:08.967 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-26T00:23:08.967 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-26T00:23:08.967 INFO:tasks.workunit.client.0.smithi129.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-03-26T00:23:08.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6157: TEST_corrupt_snapset_scrub_rep: jq .inconsistents 2024-03-26T00:23:08.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6157: TEST_corrupt_snapset_scrub_rep: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-26T00:23:09.002 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6158: TEST_corrupt_snapset_scrub_rep: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-26T00:23:09.002 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-26T00:23:09.005 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6159: TEST_corrupt_snapset_scrub_rep: test no = yes 2024-03-26T00:23:09.005 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6164: TEST_corrupt_snapset_scrub_rep: test '' = yes 2024-03-26T00:23:09.005 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6169: TEST_corrupt_snapset_scrub_rep: ERRORS=0 2024-03-26T00:23:09.005 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6170: TEST_corrupt_snapset_scrub_rep: declare -a err_strings 2024-03-26T00:23:09.005 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6171: TEST_corrupt_snapset_scrub_rep: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ1:head : snapset inconsistent' 2024-03-26T00:23:09.005 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6172: TEST_corrupt_snapset_scrub_rep: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ2:head : snapset inconsistent' 2024-03-26T00:23:09.005 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6173: TEST_corrupt_snapset_scrub_rep: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 [0-9]*:.*:::ROBJ1:1 : is an unexpected clone' 2024-03-26T00:23:09.005 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6174: TEST_corrupt_snapset_scrub_rep: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-03-26T00:23:09.006 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6175: TEST_corrupt_snapset_scrub_rep: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 0 missing, 2 inconsistent objects' 2024-03-26T00:23:09.006 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6176: TEST_corrupt_snapset_scrub_rep: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 4 errors' 2024-03-26T00:23:09.006 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-03-26T00:23:09.006 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ1:head : snapset inconsistent' td/osd-scrub-repair/osd.1.log 2024-03-26T00:23:09.007 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-03-26T00:23:09.007 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ2:head : snapset inconsistent' td/osd-scrub-repair/osd.1.log 2024-03-26T00:23:09.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-03-26T00:23:09.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 [0-9]*:.*:::ROBJ1:1 : is an unexpected clone' td/osd-scrub-repair/osd.1.log 2024-03-26T00:23:09.013 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-03-26T00:23:09.013 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-03-26T00:23:09.016 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-03-26T00:23:09.016 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 0 missing, 2 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-03-26T00:23:09.018 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-03-26T00:23:09.018 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 4 errors' td/osd-scrub-repair/osd.1.log 2024-03-26T00:23:09.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6187: TEST_corrupt_snapset_scrub_rep: '[' 0 '!=' 0 ']' 2024-03-26T00:23:09.022 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6193: TEST_corrupt_snapset_scrub_rep: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2024-03-26T00:23:09.425 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'csr_pool' removed 2024-03-26T00:23:09.446 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-26T00:23:09.446 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-26T00:23:09.446 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-26T00:23:09.446 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-26T00:23:09.447 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:23:09.447 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:23:09.447 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:23:09.447 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:23:09.447 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:23:09.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:23:09.762 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-26T00:23:09.763 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-26T00:23:09.764 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-26T00:23:09.766 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-26T00:23:09.766 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-26T00:23:09.767 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-26T00:23:09.768 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:23:09.768 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-26T00:23:09.769 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-26T00:23:09.769 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:23:09.771 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-26T00:23:09.773 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-26T00:23:09.773 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-26T00:23:09.790 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-26T00:23:09.791 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:23:09.791 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:23:09.791 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-26T00:23:09.792 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-26T00:23:09.793 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-26T00:23:09.793 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-26T00:23:09.793 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-26T00:23:09.793 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-26T00:23:09.793 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-26T00:23:09.793 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-26T00:23:09.793 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-26T00:23:09.793 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-26T00:23:09.794 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:23:09.794 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:23:09.794 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:23:09.794 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:23:09.794 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:23:09.796 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:23:09.797 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-26T00:23:09.799 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-26T00:23:09.799 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-26T00:23:09.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-26T00:23:09.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-26T00:23:09.801 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-26T00:23:09.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:23:09.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-26T00:23:09.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-26T00:23:09.803 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:23:09.804 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-26T00:23:09.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-26T00:23:09.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-26T00:23:09.807 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-26T00:23:09.808 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:23:09.808 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:23:09.808 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-26T00:23:09.809 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-26T00:23:09.809 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-26T00:23:09.809 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-26T00:23:09.811 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-26T00:23:09.811 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:23:09.811 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:23:09.811 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61636 2024-03-26T00:23:09.814 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-26T00:23:09.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-26T00:23:09.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-26T00:23:09.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-26T00:23:09.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_list_missing_erasure_coded_appends td/osd-scrub-repair 2024-03-26T00:23:09.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:978: TEST_list_missing_erasure_coded_appends: list_missing_erasure_coded td/osd-scrub-repair false 2024-03-26T00:23:09.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:916: list_missing_erasure_coded: local dir=td/osd-scrub-repair 2024-03-26T00:23:09.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:917: list_missing_erasure_coded: local allow_overwrites=false 2024-03-26T00:23:09.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: list_missing_erasure_coded: local poolname=ecpool 2024-03-26T00:23:09.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:920: list_missing_erasure_coded: run_mon td/osd-scrub-repair a 2024-03-26T00:23:09.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-26T00:23:09.815 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-26T00:23:09.815 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-26T00:23:09.815 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-26T00:23:09.815 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-26T00:23:09.815 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-26T00:23:10.294 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-26T00:23:10.294 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:23:10.294 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:23:10.295 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:23:10.295 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:23:10.295 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:23:10.295 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:23:10.296 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-26T00:23:10.530 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-26T00:23:10.531 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-26T00:23:10.531 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-26T00:23:10.532 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-26T00:23:10.532 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-26T00:23:10.532 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-26T00:23:10.532 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-26T00:23:10.532 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-26T00:23:10.533 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-26T00:23:10.533 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-26T00:23:10.533 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:23:10.533 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:23:10.534 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-26T00:23:10.534 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-26T00:23:10.534 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get fsid 2024-03-26T00:23:10.652 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-26T00:23:10.652 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-26T00:23:10.652 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-26T00:23:10.652 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-26T00:23:10.652 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-26T00:23:10.653 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-26T00:23:10.653 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-26T00:23:10.653 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-26T00:23:10.654 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-26T00:23:10.654 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:23:10.654 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:23:10.655 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-26T00:23:10.655 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-26T00:23:10.655 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get mon_host 2024-03-26T00:23:10.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:921: list_missing_erasure_coded: run_mgr td/osd-scrub-repair x 2024-03-26T00:23:10.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-26T00:23:10.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-26T00:23:10.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-26T00:23:10.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-26T00:23:10.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-26T00:23:10.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-26T00:23:11.629 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-26T00:23:11.629 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:23:11.629 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:23:11.630 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:23:11.630 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:23:11.630 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:23:11.630 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:23:11.631 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-26T00:23:11.632 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-26T00:23:11.667 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: seq 0 2 2024-03-26T00:23:11.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-26T00:23:11.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 0 2024-03-26T00:23:11.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-26T00:23:11.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-26T00:23:11.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-26T00:23:11.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-26T00:23:11.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:23:11.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:23:11.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:23:11.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:23:11.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:23:11.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-26T00:23:11.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-26T00:23:11.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-26T00:23:11.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-26T00:23:11.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:23:11.671 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-26T00:23:11.671 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:23:11.671 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:23:11.671 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:23:11.671 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:23:11.671 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:23:11.672 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:23:11.673 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:23:11.673 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:23:11.673 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-26T00:23:11.674 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-26T00:23:11.674 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:23:11.674 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:23:11.674 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:23:11.674 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:23:11.674 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:23:11.674 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:23:11.674 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-26T00:23:11.674 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-26T00:23:11.674 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-26T00:23:11.674 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-26T00:23:11.674 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=03be5642-6914-4c98-b407-2f281c443d55 2024-03-26T00:23:11.675 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 03be5642-6914-4c98-b407-2f281c443d55' 2024-03-26T00:23:11.675 INFO:tasks.workunit.client.0.smithi129.stdout:add osd0 03be5642-6914-4c98-b407-2f281c443d55 2024-03-26T00:23:11.675 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-26T00:23:11.693 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBvFQJmVH5QKRAA6Le87kn6kp8TAKpcknz5Tw== 2024-03-26T00:23:11.693 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBvFQJmVH5QKRAA6Le87kn6kp8TAKpcknz5Tw=="}' 2024-03-26T00:23:11.693 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 03be5642-6914-4c98-b407-2f281c443d55 -i td/osd-scrub-repair/0/new.json 2024-03-26T00:23:11.934 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-26T00:23:11.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-26T00:23:11.951 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBvFQJmVH5QKRAA6Le87kn6kp8TAKpcknz5Tw== --osd-uuid 03be5642-6914-4c98-b407-2f281c443d55 2024-03-26T00:23:11.984 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:23:11.984+0000 7f6c07be97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:23:11.986 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:23:11.984+0000 7f6c07be97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:23:11.987 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:23:11.984+0000 7f6c07be97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:23:11.988 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:23:11.984+0000 7f6c07be97c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-26T00:23:11.988 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:23:11.984+0000 7f6c07be97c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-26T00:23:11.988 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:23:11.984+0000 7f6c07be97c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-26T00:23:12.084 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:23:12.084+0000 7f6c07be97c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-26T00:23:18.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-26T00:23:18.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-26T00:23:18.343 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd0 key to auth repository 2024-03-26T00:23:18.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-26T00:23:18.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-26T00:23:18.740 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.0 2024-03-26T00:23:18.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-26T00:23:18.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-26T00:23:18.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:23:18.741 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-26T00:23:18.741 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-26T00:23:18.772 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:23:18.767+0000 7f888c2ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:23:18.800 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:23:18.799+0000 7f888c2ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:23:18.802 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:23:18.799+0000 7f888c2ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:23:19.075 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-26T00:23:19.075 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:23:19.075 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-26T00:23:19.076 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:23:19.076 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:23:19.076 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:23:19.076 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:23:19.076 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-26T00:23:19.076 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:23:19.076 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:23:19.358 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:23:19.355+0000 7f888c2ab7c0 -1 Falling back to public interface 2024-03-26T00:23:19.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:23:20.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:23:20.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:23:20.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:23:20.388 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-26T00:23:20.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:23:20.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:23:20.693 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:23:21.694 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:23:21.694 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:23:21.694 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:23:21.695 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-26T00:23:21.695 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:23:21.695 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:23:21.932 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:23:21.927+0000 7f888c2ab7c0 -1 osd.0 0 log_to_monitors true 2024-03-26T00:23:22.049 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:23:23.051 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:23:23.051 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:23:23.051 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-26T00:23:23.051 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:23:23.052 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:23:23.052 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:23:23.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:23:24.375 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:23:24.375 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:23:24.375 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-26T00:23:24.376 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-26T00:23:24.376 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:23:24.376 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:23:24.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:23:25.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:23:25.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:23:25.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-26T00:23:25.757 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-26T00:23:25.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:23:25.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:23:26.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:23:27.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:23:27.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:23:27.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-26T00:23:27.070 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-26T00:23:27.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:23:27.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:23:27.384 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:23:28.385 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:23:28.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:23:28.386 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-26T00:23:28.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-26T00:23:28.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:23:28.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:23:28.816 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:23:29.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:23:29.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:23:29.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-26T00:23:29.817 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-26T00:23:29.818 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:23:29.818 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:23:30.137 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:23:31.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:23:31.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:23:31.139 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-26T00:23:31.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-26T00:23:31.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:23:31.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:23:31.462 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:23:32.463 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:23:32.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:23:32.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-26T00:23:32.464 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-26T00:23:32.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:23:32.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:23:32.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:23:33.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:23:33.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:23:33.861 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-26T00:23:33.862 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-26T00:23:33.862 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:23:33.862 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:23:34.159 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:23:35.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:23:35.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:23:35.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-26T00:23:35.161 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-26T00:23:35.161 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:23:35.161 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:23:35.483 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:23:36.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:23:36.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:23:36.486 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-26T00:23:36.486 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-26T00:23:36.487 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:23:36.487 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:23:37.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:23:38.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:23:38.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:23:38.391 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-26T00:23:38.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-26T00:23:38.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:23:38.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:23:39.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:23:40.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:23:40.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:23:40.232 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-26T00:23:40.233 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-26T00:23:40.233 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:23:40.233 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:23:40.985 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:23:41.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:23:41.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:23:41.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-26T00:23:41.988 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-26T00:23:41.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:23:41.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:23:42.871 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:23:43.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:23:43.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:23:43.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-26T00:23:43.874 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-26T00:23:43.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:23:43.875 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:23:44.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:23:45.455 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:23:45.455 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:23:45.456 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-26T00:23:45.456 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-26T00:23:45.456 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:23:45.456 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:23:45.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:23:46.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:23:46.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:23:46.831 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-26T00:23:46.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-26T00:23:46.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:23:46.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:23:47.255 INFO:tasks.workunit.client.0.smithi129.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/647845770,v1:127.0.0.1:6803/647845770] [v2:127.0.0.1:6804/647845770,v1:127.0.0.1:6805/647845770] exists,up 03be5642-6914-4c98-b407-2f281c443d55 2024-03-26T00:23:47.258 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:23:47.258 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:23:47.258 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:23:47.258 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-26T00:23:47.258 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 1 2024-03-26T00:23:47.258 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-26T00:23:47.258 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-26T00:23:47.258 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-26T00:23:47.259 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-26T00:23:47.259 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:23:47.259 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:23:47.259 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:23:47.259 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:23:47.259 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:23:47.259 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:23:47.259 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:23:47.259 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-26T00:23:47.259 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-26T00:23:47.259 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:23:47.259 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-26T00:23:47.259 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:23:47.260 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:23:47.260 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:23:47.260 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:23:47.260 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:23:47.260 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:23:47.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:23:47.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:23:47.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-26T00:23:47.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-26T00:23:47.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:23:47.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:23:47.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:23:47.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:23:47.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:23:47.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:23:47.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-26T00:23:47.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-26T00:23:47.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:23:47.261 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-26T00:23:47.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=89ac67d8-d404-46be-adb3-247614496301 2024-03-26T00:23:47.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 89ac67d8-d404-46be-adb3-247614496301' 2024-03-26T00:23:47.262 INFO:tasks.workunit.client.0.smithi129.stdout:add osd1 89ac67d8-d404-46be-adb3-247614496301 2024-03-26T00:23:47.262 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-26T00:23:47.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCTFQJmBNfhEBAAMMZtayzuqWTjhfkkxGdu+g== 2024-03-26T00:23:47.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCTFQJmBNfhEBAAMMZtayzuqWTjhfkkxGdu+g=="}' 2024-03-26T00:23:47.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 89ac67d8-d404-46be-adb3-247614496301 -i td/osd-scrub-repair/1/new.json 2024-03-26T00:23:47.760 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-26T00:23:47.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-26T00:23:47.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCTFQJmBNfhEBAAMMZtayzuqWTjhfkkxGdu+g== --osd-uuid 89ac67d8-d404-46be-adb3-247614496301 2024-03-26T00:23:47.816 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:23:47.815+0000 7fb38c6bc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:23:47.870 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:23:47.867+0000 7fb38c6bc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:23:47.872 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:23:47.867+0000 7fb38c6bc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:23:47.872 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:23:47.871+0000 7fb38c6bc7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-26T00:23:47.872 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:23:47.871+0000 7fb38c6bc7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-26T00:23:47.872 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:23:47.871+0000 7fb38c6bc7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-26T00:23:47.953 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:23:47.951+0000 7fb38c6bc7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-26T00:23:54.621 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-26T00:23:54.621 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-26T00:23:54.622 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-26T00:23:54.622 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd1 key to auth repository 2024-03-26T00:23:54.622 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-26T00:23:55.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-26T00:23:55.021 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.1 2024-03-26T00:23:55.022 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-26T00:23:55.022 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-26T00:23:55.022 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-26T00:23:55.022 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:23:55.054 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:23:55.051+0000 7f3c3cbc57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:23:55.056 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:23:55.051+0000 7f3c3cbc57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:23:55.058 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:23:55.055+0000 7f3c3cbc57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:23:55.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-26T00:23:55.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:23:55.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:23:55.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:23:55.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:23:55.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:23:55.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:23:55.348 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-26T00:23:55.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:23:55.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:23:55.619 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:23:55.615+0000 7f3c3cbc57c0 -1 Falling back to public interface 2024-03-26T00:23:55.682 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:23:56.684 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:23:56.684 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:23:56.685 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:23:56.685 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-26T00:23:56.685 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:23:56.685 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:23:57.048 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:23:58.049 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:23:58.050 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:23:58.050 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-26T00:23:58.050 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:23:58.051 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:23:58.051 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:23:58.334 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:23:58.331+0000 7f3c3cbc57c0 -1 osd.1 0 log_to_monitors true 2024-03-26T00:23:58.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:23:59.361 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:23:59.361 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:23:59.361 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-26T00:23:59.362 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:23:59.362 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:23:59.362 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:23:59.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:24:00.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:24:00.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:24:00.668 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-26T00:24:00.668 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-26T00:24:00.668 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:24:00.668 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:24:01.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:24:02.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:24:02.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:24:02.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-26T00:24:02.070 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-26T00:24:02.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:24:02.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:24:02.468 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:24:03.469 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:24:03.470 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:24:03.470 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-26T00:24:03.470 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-26T00:24:03.471 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:24:03.471 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:24:03.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:24:04.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:24:04.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:24:04.784 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-26T00:24:04.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-26T00:24:04.785 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:24:04.785 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:24:05.367 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:24:06.369 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:24:06.369 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:24:06.369 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-26T00:24:06.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-26T00:24:06.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:24:06.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:24:06.699 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:24:07.700 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:24:07.701 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:24:07.701 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-26T00:24:07.701 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-26T00:24:07.701 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:24:07.702 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:24:08.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:24:09.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:24:09.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:24:09.071 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-26T00:24:09.071 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-26T00:24:09.071 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:24:09.071 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:24:09.440 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:24:10.442 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:24:10.442 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:24:10.442 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-26T00:24:10.443 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-26T00:24:10.443 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:24:10.443 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:24:10.816 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:24:11.818 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:24:11.818 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:24:11.818 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-26T00:24:11.819 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-26T00:24:11.819 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:24:11.819 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:24:12.210 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:24:13.212 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:24:13.212 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:24:13.212 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-26T00:24:13.213 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-26T00:24:13.213 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:24:13.213 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:24:13.906 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:24:14.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:24:14.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:24:14.908 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-26T00:24:14.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-26T00:24:14.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:24:14.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:24:15.279 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:24:16.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:24:16.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:24:16.281 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-26T00:24:16.282 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-26T00:24:16.282 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:24:16.282 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:24:16.881 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:24:17.883 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:24:17.883 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:24:17.884 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-26T00:24:17.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-26T00:24:17.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:24:17.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:24:18.405 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:24:19.407 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:24:19.407 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:24:19.408 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-26T00:24:19.408 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-26T00:24:19.408 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:24:19.408 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:24:20.041 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:24:21.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:24:21.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:24:21.043 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-26T00:24:21.044 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-26T00:24:21.044 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:24:21.044 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:24:21.551 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:24:22.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:24:22.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:24:22.553 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-26T00:24:22.554 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-26T00:24:22.554 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:24:22.554 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:24:23.215 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:24:24.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:24:24.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:24:24.217 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-26T00:24:24.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-26T00:24:24.218 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:24:24.218 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:24:24.624 INFO:tasks.workunit.client.0.smithi129.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4023645321,v1:127.0.0.1:6811/4023645321] [v2:127.0.0.1:6812/4023645321,v1:127.0.0.1:6813/4023645321] exists,up 89ac67d8-d404-46be-adb3-247614496301 2024-03-26T00:24:24.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:24:24.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:24:24.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:24:24.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-26T00:24:24.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 2 2024-03-26T00:24:24.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-26T00:24:24.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-26T00:24:24.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-26T00:24:24.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-26T00:24:24.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-26T00:24:24.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:24:24.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:24:24.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:24:24.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:24:24.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-26T00:24:24.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-26T00:24:24.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-26T00:24:24.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-26T00:24:24.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:24:24.627 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-26T00:24:24.627 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:24:24.627 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:24:24.627 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:24:24.628 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:24:24.628 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:24:24.629 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:24:24.629 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:24:24.629 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:24:24.629 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-26T00:24:24.629 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-26T00:24:24.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:24:24.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:24:24.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:24:24.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:24:24.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:24:24.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:24:24.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-26T00:24:24.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-26T00:24:24.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-26T00:24:24.630 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-26T00:24:24.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9a407378-07ba-4d0a-8e03-77f5c2c1c516 2024-03-26T00:24:24.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 9a407378-07ba-4d0a-8e03-77f5c2c1c516' 2024-03-26T00:24:24.631 INFO:tasks.workunit.client.0.smithi129.stdout:add osd2 9a407378-07ba-4d0a-8e03-77f5c2c1c516 2024-03-26T00:24:24.631 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-26T00:24:24.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC4FQJm/dX1JhAAgLk6jA7pSLWOde0eaQPrCQ== 2024-03-26T00:24:24.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC4FQJm/dX1JhAAgLk6jA7pSLWOde0eaQPrCQ=="}' 2024-03-26T00:24:24.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9a407378-07ba-4d0a-8e03-77f5c2c1c516 -i td/osd-scrub-repair/2/new.json 2024-03-26T00:24:25.027 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-26T00:24:25.044 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-26T00:24:25.046 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC4FQJm/dX1JhAAgLk6jA7pSLWOde0eaQPrCQ== --osd-uuid 9a407378-07ba-4d0a-8e03-77f5c2c1c516 2024-03-26T00:24:25.075 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:24:25.070+0000 7febbb0e47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:24:25.128 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:24:25.122+0000 7febbb0e47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:24:25.129 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:24:25.126+0000 7febbb0e47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:24:25.130 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:24:25.126+0000 7febbb0e47c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-26T00:24:25.130 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:24:25.126+0000 7febbb0e47c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-26T00:24:25.130 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:24:25.126+0000 7febbb0e47c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-26T00:24:25.213 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:24:25.210+0000 7febbb0e47c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-26T00:24:32.245 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-26T00:24:32.245 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-26T00:24:32.246 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd2 key to auth repository 2024-03-26T00:24:32.247 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-26T00:24:32.247 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-26T00:24:32.636 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.2 2024-03-26T00:24:32.636 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-26T00:24:32.637 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-26T00:24:32.637 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-26T00:24:32.637 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:24:32.637 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-26T00:24:32.668 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:24:32.662+0000 7fef987687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:24:32.669 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:24:32.666+0000 7fef987687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:24:32.671 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:24:32.666+0000 7fef987687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:24:32.978 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-26T00:24:32.978 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:24:32.978 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-26T00:24:32.978 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:24:32.978 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:24:32.978 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:24:32.978 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:24:32.979 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-26T00:24:32.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:24:32.979 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:24:33.211 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:24:33.206+0000 7fef987687c0 -1 Falling back to public interface 2024-03-26T00:24:33.306 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:24:34.308 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:24:34.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:24:34.309 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-26T00:24:34.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:24:34.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:24:34.310 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:24:34.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:24:35.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:24:35.610 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:24:35.610 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-26T00:24:35.610 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:24:35.611 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:24:35.611 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:24:35.867 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:24:35.862+0000 7fef987687c0 -1 osd.2 0 log_to_monitors true 2024-03-26T00:24:35.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:24:36.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:24:36.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:24:36.930 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-26T00:24:36.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:24:36.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:24:36.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:24:37.244 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:24:38.246 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:24:38.247 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:24:38.247 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-26T00:24:38.247 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-26T00:24:38.247 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:24:38.247 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:24:38.624 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:24:39.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:24:39.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:24:39.627 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-26T00:24:39.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-26T00:24:39.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:24:39.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:24:39.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:24:40.968 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:24:40.968 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:24:40.968 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-26T00:24:40.969 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-26T00:24:40.969 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:24:40.969 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:24:41.368 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:24:42.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:24:42.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:24:42.370 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-26T00:24:42.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-26T00:24:42.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:24:42.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:24:42.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:24:43.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:24:43.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:24:43.834 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-26T00:24:43.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-26T00:24:43.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:24:43.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:24:44.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:24:45.155 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:24:45.155 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:24:45.155 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-26T00:24:45.156 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-26T00:24:45.156 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:24:45.156 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:24:45.471 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:24:46.472 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:24:46.472 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:24:46.473 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-26T00:24:46.473 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-26T00:24:46.473 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:24:46.473 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:24:46.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:24:47.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:24:47.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:24:47.834 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-26T00:24:47.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-26T00:24:47.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:24:47.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:24:48.154 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:24:49.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:24:49.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:24:49.153 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-26T00:24:49.154 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-26T00:24:49.154 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:24:49.154 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:24:49.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:24:50.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:24:50.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:24:50.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-26T00:24:50.460 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-26T00:24:50.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:24:50.461 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:24:51.110 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:24:52.111 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:24:52.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:24:52.112 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-26T00:24:52.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-26T00:24:52.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:24:52.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:24:52.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:24:53.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:24:53.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:24:53.911 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-26T00:24:53.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-26T00:24:53.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:24:53.912 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:24:54.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:24:55.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:24:55.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:24:55.757 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-26T00:24:55.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-26T00:24:55.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:24:55.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:24:56.318 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:24:57.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:24:57.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:24:57.320 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-26T00:24:57.320 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-26T00:24:57.320 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:24:57.320 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:24:57.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:24:58.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:24:58.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:24:58.967 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-26T00:24:58.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-26T00:24:58.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:24:58.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:24:59.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:25:00.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:25:00.529 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:25:00.529 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-26T00:25:00.529 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-26T00:25:00.530 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:25:00.530 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:25:00.858 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:25:01.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:25:01.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:25:01.860 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-26T00:25:01.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-26T00:25:01.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:25:01.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:25:02.170 INFO:tasks.workunit.client.0.smithi129.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3377701691,v1:127.0.0.1:6819/3377701691] [v2:127.0.0.1:6820/3377701691,v1:127.0.0.1:6821/3377701691] exists,up 9a407378-07ba-4d0a-8e03-77f5c2c1c516 2024-03-26T00:25:02.171 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:25:02.171 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:25:02.171 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:25:02.171 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:925: list_missing_erasure_coded: create_rbd_pool 2024-03-26T00:25:02.171 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-26T00:25:02.480 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'rbd' does not exist 2024-03-26T00:25:02.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-26T00:25:02.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-26T00:25:02.923 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'rbd' created 2024-03-26T00:25:02.945 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-26T00:25:03.947 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-26T00:25:06.358 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: list_missing_erasure_coded: wait_for_clean 2024-03-26T00:25:06.358 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:25:06.358 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:25:06.358 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:25:06.359 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:25:06.359 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:25:06.359 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:25:06.359 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:25:06.359 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:25:06.360 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:25:06.495 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:25:06.495 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:25:06.495 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:25:06.495 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:25:06.495 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:25:06.496 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:25:06.819 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:25:06.819 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:25:06.819 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-26T00:25:06.819 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:25:06.819 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:25:06.819 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:25:06.970 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836501 2024-03-26T00:25:06.971 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836501 2024-03-26T00:25:06.971 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501' 2024-03-26T00:25:06.971 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:25:06.971 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:25:07.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672976 2024-03-26T00:25:07.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672976 2024-03-26T00:25:07.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-42949672976' 2024-03-26T00:25:07.120 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:25:07.120 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:25:07.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509447 2024-03-26T00:25:07.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509447 2024-03-26T00:25:07.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-42949672976 2-64424509447' 2024-03-26T00:25:07.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:25:07.262 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836501 2024-03-26T00:25:07.262 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:25:07.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:25:07.264 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836501 2024-03-26T00:25:07.264 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:25:07.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836501 2024-03-26T00:25:07.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-03-26T00:25:07.266 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 21474836501 2024-03-26T00:25:07.266 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:25:07.573 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836501 2024-03-26T00:25:07.573 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:25:08.575 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:25:08.576 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:25:08.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836501 2024-03-26T00:25:08.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:25:08.897 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672976 2024-03-26T00:25:08.897 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:25:08.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:25:08.899 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672976 2024-03-26T00:25:08.900 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:25:08.901 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672976 2024-03-26T00:25:08.901 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 42949672976 2024-03-26T00:25:08.902 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-03-26T00:25:08.902 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:25:09.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672976 2024-03-26T00:25:09.208 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:25:09.208 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509447 2024-03-26T00:25:09.209 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:25:09.210 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:25:09.211 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509447 2024-03-26T00:25:09.211 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:25:09.213 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509447 2024-03-26T00:25:09.213 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 64424509447 2024-03-26T00:25:09.213 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-03-26T00:25:09.213 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:25:09.544 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509447 -lt 64424509447 2024-03-26T00:25:09.545 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:25:09.545 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:25:09.545 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:25:09.925 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-26T00:25:09.925 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:25:09.926 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:25:09.926 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:25:09.926 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:25:09.926 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:25:09.926 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:25:09.926 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:25:10.259 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-26T00:25:10.259 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:25:10.260 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:25:10.260 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:25:10.638 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-26T00:25:10.638 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:25:10.639 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:25:10.639 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:928: list_missing_erasure_coded: create_ec_pool ecpool false k=2 m=1 2024-03-26T00:25:10.639 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-26T00:25:10.639 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-26T00:25:10.639 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-03-26T00:25:10.639 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-26T00:25:10.639 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-03-26T00:25:11.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-26T00:25:11.161 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-26T00:25:11.817 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'ecpool' created 2024-03-26T00:25:11.842 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-26T00:25:12.844 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-03-26T00:25:12.844 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-26T00:25:12.845 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:25:12.845 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:25:12.845 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:25:12.845 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:25:12.846 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:25:12.846 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:25:12.846 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:25:12.847 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:25:12.847 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:25:12.983 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:25:12.983 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:25:12.983 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:25:12.983 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:25:12.983 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:25:12.983 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:25:13.444 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:25:13.444 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:25:13.444 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-26T00:25:13.444 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:25:13.444 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:25:13.445 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:25:13.632 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836504 2024-03-26T00:25:13.632 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836504 2024-03-26T00:25:13.632 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504' 2024-03-26T00:25:13.632 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:25:13.632 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:25:13.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672978 2024-03-26T00:25:13.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672978 2024-03-26T00:25:13.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-42949672978' 2024-03-26T00:25:13.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:25:13.783 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:25:13.925 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509449 2024-03-26T00:25:13.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509449 2024-03-26T00:25:13.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-42949672978 2-64424509449' 2024-03-26T00:25:13.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:25:13.926 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836504 2024-03-26T00:25:13.926 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:25:13.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:25:13.928 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836504 2024-03-26T00:25:13.928 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:25:13.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836504 2024-03-26T00:25:13.929 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 21474836504 2024-03-26T00:25:13.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-03-26T00:25:13.930 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:25:14.251 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836504 2024-03-26T00:25:14.252 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:25:15.253 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:25:15.254 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:25:15.573 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836504 2024-03-26T00:25:15.574 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:25:15.575 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672978 2024-03-26T00:25:15.575 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:25:15.576 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:25:15.577 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672978 2024-03-26T00:25:15.578 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:25:15.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672978 2024-03-26T00:25:15.580 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 42949672978 2024-03-26T00:25:15.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-03-26T00:25:15.580 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:25:15.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672978 -lt 42949672978 2024-03-26T00:25:15.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:25:15.927 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509449 2024-03-26T00:25:15.927 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:25:15.928 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:25:15.929 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509449 2024-03-26T00:25:15.929 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:25:15.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509449 2024-03-26T00:25:15.931 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 64424509449 2024-03-26T00:25:15.931 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-03-26T00:25:15.931 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:25:16.289 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509449 -lt 64424509449 2024-03-26T00:25:16.290 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:25:16.291 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:25:16.291 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:25:16.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-26T00:25:16.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:25:16.660 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:25:16.660 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:25:16.660 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:25:16.660 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:25:16.660 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:25:16.660 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:25:17.009 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-26T00:25:17.010 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:25:17.010 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:25:17.011 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:25:17.367 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-26T00:25:17.367 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:25:17.367 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:25:17.367 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-26T00:25:17.367 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:931: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ0 2024-03-26T00:25:17.368 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-26T00:25:17.368 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-26T00:25:17.368 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ0 2024-03-26T00:25:17.368 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-26T00:25:17.368 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-26T00:25:17.368 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-26T00:25:17.891 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-26T00:25:17.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-26T00:25:18.435 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-26T00:25:18.454 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-26T00:25:18.454 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-26T00:25:18.455 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ0 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:25:18.801 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: get_osds ecpool MOBJ0 2024-03-26T00:25:18.801 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-26T00:25:18.801 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ0 2024-03-26T00:25:18.802 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ0 2024-03-26T00:25:18.802 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-26T00:25:19.112 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-03-26T00:25:19.112 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:25:19.112 INFO:tasks.workunit.client.0.smithi129.stderr:0' 2024-03-26T00:25:19.112 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-03-26T00:25:19.114 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: osds0=('2' '1' '0') 2024-03-26T00:25:19.114 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: local -a osds0 2024-03-26T00:25:19.114 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ1 2024-03-26T00:25:19.114 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-26T00:25:19.114 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-26T00:25:19.114 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ1 2024-03-26T00:25:19.114 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-26T00:25:19.114 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-26T00:25:19.114 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-26T00:25:19.547 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-26T00:25:19.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-26T00:25:20.175 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-26T00:25:20.195 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-26T00:25:20.195 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-26T00:25:20.196 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ1 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:25:20.516 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: get_osds ecpool MOBJ1 2024-03-26T00:25:20.516 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-26T00:25:20.516 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ1 2024-03-26T00:25:20.517 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ1 2024-03-26T00:25:20.517 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-26T00:25:20.871 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-03-26T00:25:20.872 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:25:20.872 INFO:tasks.workunit.client.0.smithi129.stderr:0' 2024-03-26T00:25:20.872 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-03-26T00:25:20.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: osds1=('2' '1' '0') 2024-03-26T00:25:20.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: local -a osds1 2024-03-26T00:25:20.872 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: seq 0 2 2024-03-26T00:25:20.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-26T00:25:20.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-26T00:25:20.872 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:25:20.872 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:25:20.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:25:20.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:25:20.872 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:25:21.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:25:21.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-26T00:25:21.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-26T00:25:21.180 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:25:21.180 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:25:21.180 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:25:21.180 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:25:21.181 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:25:22.495 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:25:22.495 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-26T00:25:22.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-26T00:25:22.496 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:25:22.496 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:25:22.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:25:22.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:25:22.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:25:22.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:25:22.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:943: list_missing_erasure_coded: id=2 2024-03-26T00:25:22.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:944: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 MOBJ0 remove 2024-03-26T00:25:24.127 INFO:tasks.workunit.client.0.smithi129.stdout:remove 0#2:a9030505:::MOBJ0:head# 2024-03-26T00:25:24.663 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: list_missing_erasure_coded: id=1 2024-03-26T00:25:24.663 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:947: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ0 remove 2024-03-26T00:25:25.939 INFO:tasks.workunit.client.0.smithi129.stdout:remove 1#2:a9030505:::MOBJ0:head# 2024-03-26T00:25:26.475 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:950: list_missing_erasure_coded: id=1 2024-03-26T00:25:26.475 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:951: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ1 remove 2024-03-26T00:25:27.384 INFO:tasks.workunit.client.0.smithi129.stdout:remove 1#2:8dd16d55:::MOBJ1:head# 2024-03-26T00:25:27.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: id=0 2024-03-26T00:25:27.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:954: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 MOBJ1 remove 2024-03-26T00:25:29.271 INFO:tasks.workunit.client.0.smithi129.stdout:remove 2#2:8dd16d55:::MOBJ1:head# 2024-03-26T00:25:29.806 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: seq 0 2 2024-03-26T00:25:29.808 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-26T00:25:29.808 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 0 2024-03-26T00:25:29.809 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:25:29.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:25:29.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-26T00:25:29.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:25:29.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:25:29.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:25:29.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:25:29.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:25:29.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:25:29.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-26T00:25:29.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-26T00:25:29.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:25:29.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:25:29.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:25:29.812 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:25:29.812 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:25:29.813 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:25:29.813 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:25:29.813 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:25:29.813 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:25:29.813 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:25:29.813 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:25:29.813 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:25:29.813 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:25:29.813 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:25:29.813 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:25:29.813 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:25:29.813 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:25:29.813 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:25:29.813 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:25:29.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:25:29.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-26T00:25:29.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-26T00:25:29.814 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.0 2024-03-26T00:25:29.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:25:29.814 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-26T00:25:29.815 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-26T00:25:29.815 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:25:29.815 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:25:29.816 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:25:29.846 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:25:29.845+0000 7f01b58c87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:25:29.849 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:25:29.849+0000 7f01b58c87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:25:29.855 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:25:29.849+0000 7f01b58c87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:25:30.141 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-26T00:25:30.141 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:25:30.141 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-26T00:25:30.141 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:25:30.141 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:25:30.141 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:25:30.141 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:25:30.141 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:25:30.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:25:30.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:25:30.387 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:25:30.385+0000 7f01b58c87c0 -1 Falling back to public interface 2024-03-26T00:25:30.502 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:25:31.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:25:31.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:25:31.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:25:31.504 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:25:31.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:25:31.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:25:31.812 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:25:32.815 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:25:32.815 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:25:32.815 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:25:32.815 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:25:32.815 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:25:32.816 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:25:33.190 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:25:33.212 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:25:33.212+0000 7f01b58c87c0 -1 osd.0 33 log_to_monitors true 2024-03-26T00:25:34.192 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:25:34.192 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:25:34.192 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:25:34.192 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:25:34.192 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:25:34.193 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:25:34.530 INFO:tasks.workunit.client.0.smithi129.stderr:osd.0 up in weight 1 up_from 40 up_thru 40 down_at 34 last_clean_interval [5,33) [v2:127.0.0.1:6802/2474251354,v1:127.0.0.1:6803/2474251354] [v2:127.0.0.1:6804/2474251354,v1:127.0.0.1:6805/2474251354] exists,up 03be5642-6914-4c98-b407-2f281c443d55 2024-03-26T00:25:34.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:25:34.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:25:34.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:25:34.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-26T00:25:34.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 1 2024-03-26T00:25:34.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:25:34.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:25:34.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-26T00:25:34.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:25:34.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:25:34.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:25:34.533 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:25:34.533 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:25:34.533 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:25:34.533 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:25:34.533 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:25:34.533 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:25:34.533 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:25:34.533 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:25:34.533 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:25:34.533 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:25:34.533 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:25:34.535 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:25:34.535 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:25:34.535 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:25:34.535 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:25:34.536 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:25:34.536 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:25:34.536 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:25:34.536 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:25:34.536 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:25:34.536 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:25:34.536 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:25:34.536 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:25:34.536 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:25:34.536 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:25:34.536 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:25:34.536 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-26T00:25:34.537 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.1 2024-03-26T00:25:34.537 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:25:34.537 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-26T00:25:34.537 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-26T00:25:34.538 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:25:34.538 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:25:34.539 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:25:34.562 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:25:34.560+0000 7f8077ebb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:25:34.589 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:25:34.588+0000 7f8077ebb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:25:34.591 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:25:34.588+0000 7f8077ebb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:25:34.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-26T00:25:34.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:25:34.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:25:34.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:25:34.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:25:34.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:25:34.860 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:25:34.860 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:25:34.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:25:34.861 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:25:35.119 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:25:35.116+0000 7f8077ebb7c0 -1 Falling back to public interface 2024-03-26T00:25:35.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:25:36.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:25:36.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:25:36.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:25:36.177 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:25:36.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:25:36.177 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:25:36.487 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:25:37.488 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:25:37.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:25:37.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:25:37.489 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:25:37.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:25:37.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:25:37.722 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:25:37.720+0000 7f8077ebb7c0 -1 osd.1 34 log_to_monitors true 2024-03-26T00:25:37.846 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:25:38.848 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:25:38.848 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:25:38.848 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:25:38.849 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:25:38.849 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:25:38.849 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:25:39.272 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 35 last_clean_interval [10,34) [v2:127.0.0.1:6810/1001564731,v1:127.0.0.1:6811/1001564731] [v2:127.0.0.1:6812/1001564731,v1:127.0.0.1:6813/1001564731] exists,up 89ac67d8-d404-46be-adb3-247614496301 2024-03-26T00:25:39.272 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:25:39.272 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:25:39.272 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:25:39.272 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-26T00:25:39.272 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 2 2024-03-26T00:25:39.272 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:25:39.272 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:25:39.272 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-26T00:25:39.272 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:25:39.272 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-26T00:25:39.273 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:25:39.273 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:25:39.273 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:25:39.273 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:25:39.273 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-26T00:25:39.273 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-26T00:25:39.273 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:25:39.273 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:25:39.273 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:25:39.275 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:25:39.275 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:25:39.275 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:25:39.275 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:25:39.275 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:25:39.276 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:25:39.276 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:25:39.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:25:39.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:25:39.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:25:39.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:25:39.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:25:39.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:25:39.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:25:39.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:25:39.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:25:39.277 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:25:39.277 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-26T00:25:39.277 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-26T00:25:39.277 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.2 2024-03-26T00:25:39.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:25:39.278 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-26T00:25:39.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-26T00:25:39.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:25:39.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:25:39.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:25:39.314 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:25:39.312+0000 7f9942dbd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:25:39.447 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:25:39.444+0000 7f9942dbd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:25:39.449 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:25:39.448+0000 7f9942dbd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:25:39.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-26T00:25:39.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:25:39.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-26T00:25:39.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:25:39.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:25:39.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:25:39.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:25:39.711 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:25:39.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:25:39.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:25:39.987 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:25:39.984+0000 7f9942dbd7c0 -1 Falling back to public interface 2024-03-26T00:25:40.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:25:41.029 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:25:41.029 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:25:41.029 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:25:41.029 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:25:41.029 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:25:41.030 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:25:41.334 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:25:42.336 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:25:42.336 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:25:42.336 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:25:42.336 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:25:42.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:25:42.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:25:42.658 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:25:42.911 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:25:42.908+0000 7f9942dbd7c0 -1 osd.2 37 log_to_monitors true 2024-03-26T00:25:43.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:25:43.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:25:43.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:25:43.661 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:25:43.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:25:43.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:25:44.092 INFO:tasks.workunit.client.0.smithi129.stderr:osd.2 up in weight 1 up_from 47 up_thru 47 down_at 38 last_clean_interval [15,37) [v2:127.0.0.1:6818/835617724,v1:127.0.0.1:6819/835617724] [v2:127.0.0.1:6820/835617724,v1:127.0.0.1:6821/835617724] exists,up 9a407378-07ba-4d0a-8e03-77f5c2c1c516 2024-03-26T00:25:44.092 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:25:44.092 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:25:44.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:25:44.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:960: list_missing_erasure_coded: create_rbd_pool 2024-03-26T00:25:44.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-26T00:25:44.818 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'rbd' removed 2024-03-26T00:25:44.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-26T00:25:44.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-26T00:25:45.338 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'rbd' created 2024-03-26T00:25:45.358 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-26T00:25:46.360 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-26T00:25:48.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:961: list_missing_erasure_coded: wait_for_clean 2024-03-26T00:25:48.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:25:48.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:25:48.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:25:48.387 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:25:48.388 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:25:48.388 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:25:48.389 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:25:48.389 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:25:48.389 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:25:48.531 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:25:48.531 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:25:48.531 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:25:48.531 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:25:48.531 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:25:48.531 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:25:48.877 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:25:48.877 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:25:48.877 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-26T00:25:48.878 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:25:48.878 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:25:48.878 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:25:49.028 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691845 2024-03-26T00:25:49.028 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691845 2024-03-26T00:25:49.028 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691845' 2024-03-26T00:25:49.028 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:25:49.028 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:25:49.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=184683593732 2024-03-26T00:25:49.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 184683593732 2024-03-26T00:25:49.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691845 1-184683593732' 2024-03-26T00:25:49.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:25:49.175 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:25:49.312 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=201863462915 2024-03-26T00:25:49.313 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 201863462915 2024-03-26T00:25:49.313 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691845 1-184683593732 2-201863462915' 2024-03-26T00:25:49.313 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:25:49.313 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691845 2024-03-26T00:25:49.313 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:25:49.314 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:25:49.315 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691845 2024-03-26T00:25:49.315 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:25:49.316 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691845 2024-03-26T00:25:49.316 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691845' 2024-03-26T00:25:49.317 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 171798691845 2024-03-26T00:25:49.317 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:25:49.625 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691843 -lt 171798691845 2024-03-26T00:25:49.625 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:25:50.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:25:50.628 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:25:50.994 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691845 -lt 171798691845 2024-03-26T00:25:50.994 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:25:50.996 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-184683593732 2024-03-26T00:25:50.996 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:25:50.997 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:25:50.998 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-184683593732 2024-03-26T00:25:50.998 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:25:51.000 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=184683593732 2024-03-26T00:25:51.000 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 184683593732' 2024-03-26T00:25:51.000 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 184683593732 2024-03-26T00:25:51.001 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:25:51.314 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 184683593732 -lt 184683593732 2024-03-26T00:25:51.314 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:25:51.315 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-201863462915 2024-03-26T00:25:51.315 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:25:51.317 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:25:51.319 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-201863462915 2024-03-26T00:25:51.319 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:25:51.320 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=201863462915 2024-03-26T00:25:51.320 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 201863462915' 2024-03-26T00:25:51.320 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 201863462915 2024-03-26T00:25:51.321 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:25:51.646 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 201863462915 -lt 201863462915 2024-03-26T00:25:51.647 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:25:51.647 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:25:51.647 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:25:52.054 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-26T00:25:52.054 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:25:52.055 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:25:52.055 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:25:52.055 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:25:52.055 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:25:52.055 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:25:52.055 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:25:52.377 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-26T00:25:52.378 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:25:52.378 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:25:52.378 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:25:52.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-26T00:25:52.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:25:52.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:25:52.741 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: get_pg ecpool MOBJ0 2024-03-26T00:25:52.741 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-26T00:25:52.741 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=MOBJ0 2024-03-26T00:25:52.741 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool MOBJ0 2024-03-26T00:25:52.741 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-26T00:25:53.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: local pg=2.0 2024-03-26T00:25:53.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:968: list_missing_erasure_coded: repair 2.0 2024-03-26T00:25:53.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-26T00:25:53.041 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-26T00:25:53.041 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-26T00:25:53.041 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:25:53.041 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:25:53.041 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-26T00:25:53.376 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-26T00:25:11.675806+0000 2024-03-26T00:25:53.377 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-26T00:25:53.669 INFO:tasks.workunit.client.0.smithi129.stderr:instructing pg 2.0s0 on osd.2 to repair 2024-03-26T00:25:53.687 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-26T00:25:11.675806+0000 2024-03-26T00:25:53.687 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-26T00:25:53.687 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-26T00:25:11.675806+0000 2024-03-26T00:25:53.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-26T00:25:53.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-26T00:25:53.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:25:53.688 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-26T00:25:53.688 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-26T00:25:53.689 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:25:53.689 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:25:53.689 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-26T00:25:53.999 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:25:11.675806+0000 '>' 2024-03-26T00:25:11.675806+0000 2024-03-26T00:25:53.999 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:25:55.000 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:25:55.000 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:25:55.002 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-26T00:25:55.002 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-26T00:25:55.002 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:25:55.002 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:25:55.003 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-26T00:25:55.322 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:25:11.675806+0000 '>' 2024-03-26T00:25:11.675806+0000 2024-03-26T00:25:55.322 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:25:56.324 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:25:56.324 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:25:56.325 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-26T00:25:56.325 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-26T00:25:56.325 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:25:56.325 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:25:56.326 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-26T00:25:56.641 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:25:11.675806+0000 '>' 2024-03-26T00:25:11.675806+0000 2024-03-26T00:25:56.641 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:25:57.643 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:25:57.643 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:25:57.643 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-26T00:25:57.643 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-26T00:25:57.644 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:25:57.644 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:25:57.644 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-26T00:25:57.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:25:11.675806+0000 '>' 2024-03-26T00:25:11.675806+0000 2024-03-26T00:25:57.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:25:58.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:25:58.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:25:58.964 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-26T00:25:58.965 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-26T00:25:58.965 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:25:58.965 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:25:58.965 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-26T00:25:59.270 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:25:54.045291+0000 '>' 2024-03-26T00:25:11.675806+0000 2024-03-26T00:25:59.270 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-26T00:25:59.271 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: seq 0 120 2024-03-26T00:25:59.273 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: for i in $(seq 0 120) 2024-03-26T00:25:59.273 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:971: list_missing_erasure_coded: '[' 0 -lt 60 ']' 2024-03-26T00:25:59.274 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: ceph pg 2.0 list_unfound 2024-03-26T00:25:59.274 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: egrep 'MOBJ0|MOBJ1' 2024-03-26T00:25:59.274 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: wc -l 2024-03-26T00:25:59.422 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: matches=2 2024-03-26T00:25:59.422 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: '[' 2 -eq 2 ']' 2024-03-26T00:25:59.422 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: break 2024-03-26T00:25:59.423 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-26T00:25:59.423 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-26T00:25:59.423 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-26T00:25:59.423 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-26T00:25:59.423 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:25:59.424 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:25:59.424 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:25:59.424 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:25:59.424 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:25:59.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:25:59.732 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-26T00:25:59.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-26T00:25:59.735 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-26T00:25:59.736 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-26T00:25:59.736 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-26T00:25:59.737 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-26T00:25:59.738 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:25:59.738 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-26T00:25:59.739 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-26T00:25:59.740 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:25:59.741 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-26T00:25:59.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-26T00:25:59.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-26T00:25:59.765 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-26T00:25:59.765 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:25:59.765 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:25:59.765 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-26T00:25:59.768 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-26T00:25:59.768 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-26T00:25:59.768 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-26T00:25:59.768 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-26T00:25:59.768 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-26T00:25:59.768 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-26T00:25:59.769 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-26T00:25:59.769 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-26T00:25:59.769 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-26T00:25:59.769 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:25:59.769 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:25:59.769 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:25:59.769 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:25:59.769 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:25:59.770 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:25:59.771 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-26T00:25:59.772 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-26T00:25:59.773 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-26T00:25:59.774 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-26T00:25:59.775 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-26T00:25:59.777 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-26T00:25:59.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:25:59.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-26T00:25:59.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-26T00:25:59.778 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:25:59.778 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-26T00:25:59.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-26T00:25:59.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-26T00:25:59.782 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-26T00:25:59.783 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:25:59.783 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:25:59.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-26T00:25:59.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-26T00:25:59.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-26T00:25:59.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-26T00:25:59.784 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-26T00:25:59.784 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:25:59.785 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:25:59.785 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61636 2024-03-26T00:25:59.786 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-26T00:25:59.788 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-26T00:25:59.788 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-26T00:25:59.788 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-26T00:25:59.788 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_list_missing_erasure_coded_overwrites td/osd-scrub-repair 2024-03-26T00:25:59.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:982: TEST_list_missing_erasure_coded_overwrites: '[' true = true ']' 2024-03-26T00:25:59.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:983: TEST_list_missing_erasure_coded_overwrites: list_missing_erasure_coded td/osd-scrub-repair true 2024-03-26T00:25:59.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:916: list_missing_erasure_coded: local dir=td/osd-scrub-repair 2024-03-26T00:25:59.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:917: list_missing_erasure_coded: local allow_overwrites=true 2024-03-26T00:25:59.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: list_missing_erasure_coded: local poolname=ecpool 2024-03-26T00:25:59.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:920: list_missing_erasure_coded: run_mon td/osd-scrub-repair a 2024-03-26T00:25:59.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-26T00:25:59.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-26T00:25:59.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-26T00:25:59.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-26T00:25:59.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-26T00:25:59.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-26T00:26:00.263 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-26T00:26:00.264 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:26:00.264 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:26:00.264 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:26:00.264 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:26:00.265 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:26:00.265 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:26:00.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-26T00:26:00.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-26T00:26:00.510 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-26T00:26:00.510 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-26T00:26:00.510 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-26T00:26:00.510 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-26T00:26:00.511 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-26T00:26:00.511 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-26T00:26:00.511 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-26T00:26:00.511 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-26T00:26:00.512 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-26T00:26:00.512 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:26:00.512 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:26:00.512 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-26T00:26:00.512 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-26T00:26:00.512 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get fsid 2024-03-26T00:26:00.624 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-26T00:26:00.624 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-26T00:26:00.624 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-26T00:26:00.625 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-26T00:26:00.625 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-26T00:26:00.625 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-26T00:26:00.625 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-26T00:26:00.625 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-26T00:26:00.626 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-26T00:26:00.626 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:26:00.626 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:26:00.627 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-26T00:26:00.627 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-26T00:26:00.627 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get mon_host 2024-03-26T00:26:00.739 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:921: list_missing_erasure_coded: run_mgr td/osd-scrub-repair x 2024-03-26T00:26:00.739 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-26T00:26:00.739 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-26T00:26:00.739 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-26T00:26:00.739 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-26T00:26:00.739 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-26T00:26:00.739 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-26T00:26:01.657 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-26T00:26:01.658 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:26:01.658 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:26:01.658 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:26:01.658 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:26:01.659 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:26:01.659 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:26:01.659 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-26T00:26:01.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-26T00:26:01.695 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: seq 0 2 2024-03-26T00:26:01.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-26T00:26:01.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 0 2024-03-26T00:26:01.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-26T00:26:01.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-26T00:26:01.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-26T00:26:01.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-26T00:26:01.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:26:01.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:26:01.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:26:01.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:26:01.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:26:01.698 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-26T00:26:01.699 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-26T00:26:01.699 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-26T00:26:01.699 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-26T00:26:01.699 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:26:01.699 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-26T00:26:01.699 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:26:01.699 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:26:01.699 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:26:01.699 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:26:01.699 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:26:01.699 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:26:01.701 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:26:01.701 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:26:01.701 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-26T00:26:01.701 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-26T00:26:01.702 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:26:01.702 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:26:01.702 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:26:01.702 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:26:01.702 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:26:01.702 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:26:01.702 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-26T00:26:01.702 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-26T00:26:01.702 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-26T00:26:01.702 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-26T00:26:01.702 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7a41ffd6-b6c2-499c-9dd7-7d55c626357c 2024-03-26T00:26:01.702 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 7a41ffd6-b6c2-499c-9dd7-7d55c626357c' 2024-03-26T00:26:01.703 INFO:tasks.workunit.client.0.smithi129.stdout:add osd0 7a41ffd6-b6c2-499c-9dd7-7d55c626357c 2024-03-26T00:26:01.703 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-26T00:26:01.720 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAZFgJmoK7mKhAAhR3IA9xe7+sWt3az0ZpGcA== 2024-03-26T00:26:01.720 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAZFgJmoK7mKhAAhR3IA9xe7+sWt3az0ZpGcA=="}' 2024-03-26T00:26:01.720 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7a41ffd6-b6c2-499c-9dd7-7d55c626357c -i td/osd-scrub-repair/0/new.json 2024-03-26T00:26:02.053 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-26T00:26:02.072 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-26T00:26:02.073 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAZFgJmoK7mKhAAhR3IA9xe7+sWt3az0ZpGcA== --osd-uuid 7a41ffd6-b6c2-499c-9dd7-7d55c626357c 2024-03-26T00:26:02.107 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:26:02.104+0000 7f5bf866b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:26:02.138 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:26:02.136+0000 7f5bf866b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:26:02.139 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:26:02.136+0000 7f5bf866b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:26:02.140 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:26:02.136+0000 7f5bf866b7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-26T00:26:02.140 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:26:02.136+0000 7f5bf866b7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-26T00:26:02.140 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:26:02.136+0000 7f5bf866b7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-26T00:26:02.245 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:26:02.244+0000 7f5bf866b7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-26T00:26:10.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-26T00:26:10.218 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-26T00:26:10.219 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd0 key to auth repository 2024-03-26T00:26:10.220 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-26T00:26:10.220 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-26T00:26:10.613 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.0 2024-03-26T00:26:10.614 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-26T00:26:10.614 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:26:10.614 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-26T00:26:10.614 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-26T00:26:10.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-26T00:26:10.646 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:26:10.644+0000 7f64225037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:26:10.693 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:26:10.692+0000 7f64225037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:26:10.695 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:26:10.692+0000 7f64225037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:26:10.925 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-26T00:26:10.925 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:26:10.925 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-26T00:26:10.925 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:26:10.925 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:26:10.925 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:26:10.925 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-26T00:26:10.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:26:10.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:26:10.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:26:11.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:26:11.266 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:26:11.264+0000 7f64225037c0 -1 Falling back to public interface 2024-03-26T00:26:12.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:26:12.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:26:12.231 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-26T00:26:12.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:26:12.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:26:12.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:26:12.536 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:26:13.538 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:26:13.538 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:26:13.538 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-26T00:26:13.539 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:26:13.539 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:26:13.539 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:26:13.852 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:26:14.215 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:26:14.212+0000 7f64225037c0 -1 osd.0 0 log_to_monitors true 2024-03-26T00:26:14.853 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:26:14.854 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:26:14.854 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-26T00:26:14.854 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:26:14.855 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:26:14.855 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:26:15.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:26:16.338 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:26:16.338 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:26:16.338 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-26T00:26:16.339 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-26T00:26:16.339 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:26:16.339 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:26:16.652 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:26:17.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:26:17.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:26:17.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-26T00:26:17.655 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-26T00:26:17.656 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:26:17.656 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:26:18.018 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:26:19.020 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:26:19.020 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:26:19.021 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-26T00:26:19.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-26T00:26:19.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:26:19.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:26:19.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:26:20.519 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:26:20.519 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:26:20.519 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-26T00:26:20.520 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-26T00:26:20.520 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:26:20.520 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:26:20.844 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:26:21.846 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:26:21.846 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:26:21.846 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-26T00:26:21.846 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-26T00:26:21.847 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:26:21.847 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:26:22.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:26:23.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:26:23.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:26:23.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-26T00:26:23.350 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-26T00:26:23.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:26:23.351 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:26:23.669 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:26:24.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:26:24.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:26:24.672 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-26T00:26:24.672 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-26T00:26:24.672 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:26:24.672 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:26:25.026 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:26:26.028 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:26:26.028 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:26:26.028 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-26T00:26:26.029 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-26T00:26:26.029 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:26:26.029 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:26:26.642 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:26:27.644 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:26:27.644 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:26:27.645 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-26T00:26:27.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-26T00:26:27.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:26:27.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:26:28.084 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:26:29.086 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:26:29.087 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:26:29.087 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-26T00:26:29.088 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-26T00:26:29.088 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:26:29.088 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:26:29.506 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:26:30.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:26:30.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:26:30.508 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-26T00:26:30.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-26T00:26:30.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:26:30.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:26:30.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:26:31.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:26:31.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:26:31.835 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-26T00:26:31.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-26T00:26:31.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:26:31.836 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:26:32.315 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:26:33.317 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:26:33.317 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:26:33.318 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-26T00:26:33.318 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-26T00:26:33.318 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:26:33.318 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:26:34.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:26:35.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:26:35.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:26:35.042 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-26T00:26:35.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-26T00:26:35.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:26:35.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:26:36.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:26:37.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:26:37.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:26:37.237 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-26T00:26:37.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-26T00:26:37.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:26:37.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:26:37.709 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:26:38.710 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:26:38.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:26:38.711 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-26T00:26:38.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-26T00:26:38.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:26:38.712 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:26:39.556 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:26:40.558 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:26:40.558 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:26:40.558 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-26T00:26:40.559 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-26T00:26:40.559 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:26:40.559 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:26:41.462 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:26:42.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:26:42.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:26:42.464 INFO:tasks.workunit.client.0.smithi129.stdout:21 2024-03-26T00:26:42.465 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-26T00:26:42.465 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:26:42.465 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:26:43.163 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:26:44.164 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:26:44.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:26:44.165 INFO:tasks.workunit.client.0.smithi129.stdout:22 2024-03-26T00:26:44.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-26T00:26:44.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:26:44.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:26:44.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:26:45.502 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:26:45.502 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:26:45.502 INFO:tasks.workunit.client.0.smithi129.stdout:23 2024-03-26T00:26:45.503 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-03-26T00:26:45.503 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:26:45.504 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:26:46.063 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:26:47.065 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:26:47.065 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:26:47.065 INFO:tasks.workunit.client.0.smithi129.stdout:24 2024-03-26T00:26:47.066 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-03-26T00:26:47.066 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:26:47.066 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:26:47.525 INFO:tasks.workunit.client.0.smithi129.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2383554882,v1:127.0.0.1:6803/2383554882] [v2:127.0.0.1:6804/2383554882,v1:127.0.0.1:6805/2383554882] exists,up 7a41ffd6-b6c2-499c-9dd7-7d55c626357c 2024-03-26T00:26:47.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:26:47.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:26:47.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:26:47.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-26T00:26:47.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 1 2024-03-26T00:26:47.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-26T00:26:47.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-26T00:26:47.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-26T00:26:47.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-26T00:26:47.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:26:47.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:26:47.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:26:47.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:26:47.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:26:47.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:26:47.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:26:47.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-26T00:26:47.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-26T00:26:47.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:26:47.528 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-26T00:26:47.528 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:26:47.528 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:26:47.530 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:26:47.531 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:26:47.531 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:26:47.531 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:26:47.531 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:26:47.531 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:26:47.531 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-26T00:26:47.531 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-26T00:26:47.531 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:26:47.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:26:47.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:26:47.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:26:47.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:26:47.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:26:47.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-26T00:26:47.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-26T00:26:47.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:26:47.532 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-26T00:26:47.533 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=736d8177-73db-4bbb-b16e-6271ae1568e6 2024-03-26T00:26:47.533 INFO:tasks.workunit.client.0.smithi129.stdout:add osd1 736d8177-73db-4bbb-b16e-6271ae1568e6 2024-03-26T00:26:47.534 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 736d8177-73db-4bbb-b16e-6271ae1568e6' 2024-03-26T00:26:47.534 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-26T00:26:47.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBHFgJmTtTJIBAAT9nvjDOxBkxHVzQJwn5wkQ== 2024-03-26T00:26:47.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBHFgJmTtTJIBAAT9nvjDOxBkxHVzQJwn5wkQ=="}' 2024-03-26T00:26:47.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 736d8177-73db-4bbb-b16e-6271ae1568e6 -i td/osd-scrub-repair/1/new.json 2024-03-26T00:26:47.984 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-26T00:26:48.002 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-26T00:26:48.003 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBHFgJmTtTJIBAAT9nvjDOxBkxHVzQJwn5wkQ== --osd-uuid 736d8177-73db-4bbb-b16e-6271ae1568e6 2024-03-26T00:26:48.035 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:26:48.031+0000 7fba9b7f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:26:48.119 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:26:48.115+0000 7fba9b7f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:26:48.121 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:26:48.119+0000 7fba9b7f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:26:48.122 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:26:48.119+0000 7fba9b7f37c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-26T00:26:48.122 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:26:48.119+0000 7fba9b7f37c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-26T00:26:48.122 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:26:48.119+0000 7fba9b7f37c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-26T00:26:48.212 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:26:48.211+0000 7fba9b7f37c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-26T00:26:56.642 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-26T00:26:56.642 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-26T00:26:56.644 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd1 key to auth repository 2024-03-26T00:26:56.644 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-26T00:26:56.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-26T00:26:57.068 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.1 2024-03-26T00:26:57.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-26T00:26:57.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-26T00:26:57.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-26T00:26:57.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:26:57.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-26T00:26:57.102 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:26:57.099+0000 7f34b16f77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:26:57.158 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:26:57.155+0000 7f34b16f77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:26:57.160 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:26:57.155+0000 7f34b16f77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:26:57.360 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-26T00:26:57.360 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:26:57.360 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:26:57.360 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:26:57.360 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:26:57.361 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:26:57.361 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-26T00:26:57.361 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:26:57.361 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:26:57.361 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:26:57.734 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:26:57.731+0000 7f34b16f77c0 -1 Falling back to public interface 2024-03-26T00:26:57.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:26:58.748 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:26:58.748 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:26:58.749 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-26T00:26:58.749 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:26:58.749 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:26:58.749 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:26:59.071 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:27:00.074 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:27:00.074 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:27:00.074 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-26T00:27:00.075 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:27:00.075 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:27:00.075 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:27:00.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:27:00.743 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:27:00.739+0000 7f34b16f77c0 -1 osd.1 0 log_to_monitors true 2024-03-26T00:27:01.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:27:01.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:27:01.415 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-26T00:27:01.416 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:27:01.416 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:27:01.416 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:27:01.785 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:27:02.790 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:27:02.791 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:27:02.791 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-26T00:27:02.792 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-26T00:27:02.792 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:27:02.792 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:27:03.221 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:27:04.223 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:27:04.223 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:27:04.223 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-26T00:27:04.224 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-26T00:27:04.224 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:27:04.224 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:27:04.680 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:27:05.682 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:27:05.682 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:27:05.683 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-26T00:27:05.683 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-26T00:27:05.683 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:27:05.683 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:27:06.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:27:07.170 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:27:07.170 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:27:07.170 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-26T00:27:07.171 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-26T00:27:07.171 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:27:07.171 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:27:07.506 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:27:08.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:27:08.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:27:08.511 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-26T00:27:08.512 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-26T00:27:08.512 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:27:08.512 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:27:08.940 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:27:09.943 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:27:09.943 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:27:09.943 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-26T00:27:09.944 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-26T00:27:09.944 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:27:09.944 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:27:10.265 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:27:11.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:27:11.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:27:11.268 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-26T00:27:11.269 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-26T00:27:11.269 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:27:11.269 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:27:11.749 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:27:12.751 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:27:12.751 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:27:12.752 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-26T00:27:12.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-26T00:27:12.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:27:12.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:27:13.125 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:27:14.127 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:27:14.127 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:27:14.127 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-26T00:27:14.128 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-26T00:27:14.128 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:27:14.128 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:27:14.450 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:27:15.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:27:15.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:27:15.453 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-26T00:27:15.453 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-26T00:27:15.454 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:27:15.454 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:27:15.968 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:27:16.970 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:27:16.970 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:27:16.970 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-26T00:27:16.971 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-26T00:27:16.971 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:27:16.971 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:27:17.339 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:27:18.340 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:27:18.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:27:18.341 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-26T00:27:18.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-26T00:27:18.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:27:18.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:27:18.658 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:27:19.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:27:19.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:27:19.660 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-26T00:27:19.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-26T00:27:19.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:27:19.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:27:20.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:27:21.092 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:27:21.092 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:27:21.092 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-26T00:27:21.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-26T00:27:21.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:27:21.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:27:21.401 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:27:22.402 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:27:22.403 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:27:22.403 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-26T00:27:22.403 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-26T00:27:22.403 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:27:22.404 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:27:23.192 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:27:24.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:27:24.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:27:24.194 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-26T00:27:24.195 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-26T00:27:24.195 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:27:24.195 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:27:24.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:27:25.921 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:27:25.921 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:27:25.922 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-26T00:27:25.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-26T00:27:25.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:27:25.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:27:26.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:27:27.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:27:27.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:27:27.761 INFO:tasks.workunit.client.0.smithi129.stdout:21 2024-03-26T00:27:27.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-26T00:27:27.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:27:27.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:27:28.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:27:29.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:27:29.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:27:29.262 INFO:tasks.workunit.client.0.smithi129.stdout:22 2024-03-26T00:27:29.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-26T00:27:29.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:27:29.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:27:29.880 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:27:30.881 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:27:30.882 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:27:30.882 INFO:tasks.workunit.client.0.smithi129.stdout:23 2024-03-26T00:27:30.882 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-03-26T00:27:30.882 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:27:30.883 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:27:31.827 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:27:32.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:27:32.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:27:32.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-03-26T00:27:32.830 INFO:tasks.workunit.client.0.smithi129.stdout:24 2024-03-26T00:27:32.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:27:32.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:27:34.001 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:27:35.003 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:27:35.003 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:27:35.003 INFO:tasks.workunit.client.0.smithi129.stdout:25 2024-03-26T00:27:35.003 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-03-26T00:27:35.004 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:27:35.004 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:27:35.343 INFO:tasks.workunit.client.0.smithi129.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2216375166,v1:127.0.0.1:6811/2216375166] [v2:127.0.0.1:6812/2216375166,v1:127.0.0.1:6813/2216375166] exists,up 736d8177-73db-4bbb-b16e-6271ae1568e6 2024-03-26T00:27:35.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:27:35.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:27:35.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:27:35.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-26T00:27:35.347 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 2 2024-03-26T00:27:35.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-26T00:27:35.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-26T00:27:35.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-26T00:27:35.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-26T00:27:35.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-26T00:27:35.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:27:35.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:27:35.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:27:35.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:27:35.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-26T00:27:35.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-26T00:27:35.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-26T00:27:35.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-26T00:27:35.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:27:35.349 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-26T00:27:35.349 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:27:35.349 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:27:35.349 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:27:35.349 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:27:35.349 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:27:35.349 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:27:35.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:27:35.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:27:35.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-26T00:27:35.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-26T00:27:35.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:27:35.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:27:35.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:27:35.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:27:35.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:27:35.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:27:35.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-26T00:27:35.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-26T00:27:35.350 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-26T00:27:35.351 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-26T00:27:35.351 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c48eb426-1303-426c-b06d-e79cefe3c319 2024-03-26T00:27:35.351 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 c48eb426-1303-426c-b06d-e79cefe3c319' 2024-03-26T00:27:35.352 INFO:tasks.workunit.client.0.smithi129.stdout:add osd2 c48eb426-1303-426c-b06d-e79cefe3c319 2024-03-26T00:27:35.352 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-26T00:27:35.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB3FgJmNYkXFhAAZ6vrTrf/5ojKK2Aa7U8ChA== 2024-03-26T00:27:35.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB3FgJmNYkXFhAAZ6vrTrf/5ojKK2Aa7U8ChA=="}' 2024-03-26T00:27:35.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c48eb426-1303-426c-b06d-e79cefe3c319 -i td/osd-scrub-repair/2/new.json 2024-03-26T00:27:35.841 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-26T00:27:35.862 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-26T00:27:35.864 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB3FgJmNYkXFhAAZ6vrTrf/5ojKK2Aa7U8ChA== --osd-uuid c48eb426-1303-426c-b06d-e79cefe3c319 2024-03-26T00:27:35.898 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:27:35.898+0000 7f1694ff07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:27:35.977 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:27:35.974+0000 7f1694ff07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:27:35.979 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:27:35.974+0000 7f1694ff07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:27:35.979 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:27:35.974+0000 7f1694ff07c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-26T00:27:35.980 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:27:35.974+0000 7f1694ff07c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-26T00:27:35.980 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:27:35.974+0000 7f1694ff07c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-26T00:27:36.076 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:27:36.074+0000 7f1694ff07c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-26T00:27:44.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-26T00:27:44.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-26T00:27:44.561 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd2 key to auth repository 2024-03-26T00:27:44.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-26T00:27:44.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-26T00:27:44.942 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.2 2024-03-26T00:27:44.942 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-26T00:27:44.943 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:27:44.943 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-26T00:27:44.943 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-26T00:27:44.943 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-26T00:27:44.974 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:27:44.973+0000 7f57d20fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:27:44.995 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:27:44.993+0000 7f57d20fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:27:44.996 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:27:44.997+0000 7f57d20fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:27:45.273 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-26T00:27:45.273 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:27:45.273 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-26T00:27:45.273 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:27:45.273 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:27:45.273 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:27:45.273 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:27:45.274 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-26T00:27:45.274 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:27:45.274 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:27:45.555 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:27:45.553+0000 7f57d20fe7c0 -1 Falling back to public interface 2024-03-26T00:27:45.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:27:46.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:27:46.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:27:46.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:27:46.580 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-26T00:27:46.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:27:46.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:27:46.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:27:47.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:27:47.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:27:47.911 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-26T00:27:47.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:27:47.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:27:47.912 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:27:48.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:27:48.472 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:27:48.469+0000 7f57d20fe7c0 -1 osd.2 0 log_to_monitors true 2024-03-26T00:27:49.219 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:27:49.219 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:27:49.219 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-26T00:27:49.220 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:27:49.220 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:27:49.220 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:27:49.624 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:27:50.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:27:50.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:27:50.626 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-26T00:27:50.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-26T00:27:50.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:27:50.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:27:51.037 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:27:52.039 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:27:52.039 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:27:52.039 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-26T00:27:52.040 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-26T00:27:52.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:27:52.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:27:52.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:27:53.462 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:27:53.462 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:27:53.462 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-26T00:27:53.463 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-26T00:27:53.463 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:27:53.463 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:27:53.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:27:54.894 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:27:54.894 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:27:54.894 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-26T00:27:54.895 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-26T00:27:54.895 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:27:54.895 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:27:55.306 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:27:56.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:27:56.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:27:56.308 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-26T00:27:56.308 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-26T00:27:56.308 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:27:56.308 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:27:56.641 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:27:57.642 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:27:57.642 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:27:57.642 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-26T00:27:57.643 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-26T00:27:57.643 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:27:57.643 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:27:58.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:27:59.115 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:27:59.115 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:27:59.115 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-26T00:27:59.116 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-26T00:27:59.116 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:27:59.116 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:27:59.624 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:28:00.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:28:00.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:28:00.626 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-26T00:28:00.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-26T00:28:00.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:28:00.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:28:00.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:28:01.954 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:28:01.954 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:28:01.955 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-26T00:28:01.955 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-26T00:28:01.955 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:28:01.955 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:28:02.269 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:28:03.271 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:28:03.271 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:28:03.271 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-26T00:28:03.272 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-26T00:28:03.272 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:28:03.272 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:28:03.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:28:04.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:28:04.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:28:04.805 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-26T00:28:04.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-26T00:28:04.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:28:04.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:28:05.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:28:06.114 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:28:06.114 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:28:06.115 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-26T00:28:06.115 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-26T00:28:06.115 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:28:06.115 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:28:06.429 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:28:07.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:28:07.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:28:07.434 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-26T00:28:07.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-26T00:28:07.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:28:07.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:28:07.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:28:08.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:28:08.875 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:28:08.875 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-26T00:28:08.875 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-26T00:28:08.875 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:28:08.876 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:28:09.737 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:28:10.738 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:28:10.739 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:28:10.739 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-26T00:28:10.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-26T00:28:10.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:28:10.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:28:11.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:28:12.328 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:28:12.328 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:28:12.328 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-26T00:28:12.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-26T00:28:12.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:28:12.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:28:12.815 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:28:13.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:28:13.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:28:13.817 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-26T00:28:13.818 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-26T00:28:13.818 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:28:13.818 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:28:14.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:28:15.137 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:28:15.137 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:28:15.137 INFO:tasks.workunit.client.0.smithi129.stdout:21 2024-03-26T00:28:15.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-26T00:28:15.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:28:15.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:28:15.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:28:16.633 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:28:16.633 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:28:16.633 INFO:tasks.workunit.client.0.smithi129.stdout:22 2024-03-26T00:28:16.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-26T00:28:16.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:28:16.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:28:17.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:28:18.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:28:18.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:28:18.390 INFO:tasks.workunit.client.0.smithi129.stdout:23 2024-03-26T00:28:18.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-03-26T00:28:18.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:28:18.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:28:19.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:28:20.141 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:28:20.141 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:28:20.142 INFO:tasks.workunit.client.0.smithi129.stdout:24 2024-03-26T00:28:20.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-03-26T00:28:20.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:28:20.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:28:20.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:28:21.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:28:21.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:28:21.609 INFO:tasks.workunit.client.0.smithi129.stdout:25 2024-03-26T00:28:21.610 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-03-26T00:28:21.610 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:28:21.610 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:28:22.154 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:28:23.156 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:28:23.156 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:28:23.156 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 26 2024-03-26T00:28:23.156 INFO:tasks.workunit.client.0.smithi129.stdout:26 2024-03-26T00:28:23.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:28:23.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:28:23.473 INFO:tasks.workunit.client.0.smithi129.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2972887517,v1:127.0.0.1:6819/2972887517] [v2:127.0.0.1:6820/2972887517,v1:127.0.0.1:6821/2972887517] exists,up c48eb426-1303-426c-b06d-e79cefe3c319 2024-03-26T00:28:23.474 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:28:23.474 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:28:23.474 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:28:23.474 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:925: list_missing_erasure_coded: create_rbd_pool 2024-03-26T00:28:23.475 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-26T00:28:23.777 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'rbd' does not exist 2024-03-26T00:28:23.795 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-26T00:28:23.795 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-26T00:28:24.368 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'rbd' created 2024-03-26T00:28:24.387 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-26T00:28:25.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-26T00:28:28.961 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: list_missing_erasure_coded: wait_for_clean 2024-03-26T00:28:28.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:28:28.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:28:28.962 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:28:28.962 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:28:28.963 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:28:28.964 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:28:28.964 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:28:28.964 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:28:28.964 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:28:29.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:28:29.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:28:29.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:28:29.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:28:29.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:28:29.102 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:28:29.422 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:28:29.422 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:28:29.422 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-26T00:28:29.422 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:28:29.422 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:28:29.422 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:28:29.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836507 2024-03-26T00:28:29.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836507 2024-03-26T00:28:29.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507' 2024-03-26T00:28:29.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:28:29.567 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:28:29.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672981 2024-03-26T00:28:29.708 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672981 2024-03-26T00:28:29.708 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-42949672981' 2024-03-26T00:28:29.708 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:28:29.708 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:28:29.851 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509454 2024-03-26T00:28:29.851 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509454 2024-03-26T00:28:29.851 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-42949672981 2-64424509454' 2024-03-26T00:28:29.851 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:28:29.852 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836507 2024-03-26T00:28:29.852 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:28:29.853 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:28:29.854 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836507 2024-03-26T00:28:29.854 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:28:29.856 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836507 2024-03-26T00:28:29.856 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-03-26T00:28:29.856 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 21474836507 2024-03-26T00:28:29.856 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:28:30.185 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836507 2024-03-26T00:28:30.185 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:28:31.190 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:28:31.190 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:28:31.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836507 -lt 21474836507 2024-03-26T00:28:31.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:28:31.512 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672981 2024-03-26T00:28:31.512 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:28:31.514 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:28:31.515 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672981 2024-03-26T00:28:31.515 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:28:31.516 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672981 2024-03-26T00:28:31.516 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-03-26T00:28:31.517 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 42949672981 2024-03-26T00:28:31.517 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:28:31.837 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672981 -lt 42949672981 2024-03-26T00:28:31.837 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:28:31.838 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509454 2024-03-26T00:28:31.839 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:28:31.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:28:31.841 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509454 2024-03-26T00:28:31.841 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:28:31.842 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509454 2024-03-26T00:28:31.843 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-03-26T00:28:31.843 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 64424509454 2024-03-26T00:28:31.843 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:28:32.202 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509454 -lt 64424509454 2024-03-26T00:28:32.203 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:28:32.203 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:28:32.204 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:28:32.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-26T00:28:32.666 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:28:32.667 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:28:32.667 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:28:32.668 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:28:32.668 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:28:32.668 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:28:32.668 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:28:33.025 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-26T00:28:33.026 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:28:33.027 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:28:33.027 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:28:33.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-26T00:28:33.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:28:33.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:28:33.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:928: list_missing_erasure_coded: create_ec_pool ecpool true k=2 m=1 2024-03-26T00:28:33.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-26T00:28:33.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-26T00:28:33.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-03-26T00:28:33.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-26T00:28:33.517 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-03-26T00:28:34.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-26T00:28:34.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-26T00:28:35.330 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'ecpool' created 2024-03-26T00:28:35.356 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-26T00:28:36.358 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-03-26T00:28:36.358 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-03-26T00:28:37.188 INFO:tasks.workunit.client.0.smithi129.stderr:set pool 2 allow_ec_overwrites to true 2024-03-26T00:28:37.211 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-26T00:28:37.211 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:28:37.211 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:28:37.211 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:28:37.211 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:28:37.212 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:28:37.212 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:28:37.213 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:28:37.213 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:28:37.213 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:28:37.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:28:37.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:28:37.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:28:37.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:28:37.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:28:37.343 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:28:37.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:28:37.800 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:28:37.800 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-26T00:28:37.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:28:37.801 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:28:37.801 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:28:37.951 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836510 2024-03-26T00:28:37.951 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836510 2024-03-26T00:28:37.951 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510' 2024-03-26T00:28:37.951 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:28:37.951 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:28:38.164 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672984 2024-03-26T00:28:38.164 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672984 2024-03-26T00:28:38.164 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510 1-42949672984' 2024-03-26T00:28:38.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:28:38.165 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:28:38.308 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-03-26T00:28:38.308 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-03-26T00:28:38.308 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510 1-42949672984 2-64424509456' 2024-03-26T00:28:38.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:28:38.309 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836510 2024-03-26T00:28:38.309 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:28:38.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:28:38.310 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836510 2024-03-26T00:28:38.310 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:28:38.312 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836510 2024-03-26T00:28:38.312 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-03-26T00:28:38.312 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 21474836510 2024-03-26T00:28:38.312 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:28:38.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836510 -lt 21474836510 2024-03-26T00:28:38.624 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:28:38.625 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672984 2024-03-26T00:28:38.625 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:28:38.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:28:38.629 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672984 2024-03-26T00:28:38.629 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:28:38.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672984 2024-03-26T00:28:38.630 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 42949672984 2024-03-26T00:28:38.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-03-26T00:28:38.631 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:28:38.938 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672984 -lt 42949672984 2024-03-26T00:28:38.939 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:28:38.940 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-03-26T00:28:38.940 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:28:38.941 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:28:38.942 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-03-26T00:28:38.942 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:28:38.943 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-03-26T00:28:38.944 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 64424509456 2024-03-26T00:28:38.944 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-03-26T00:28:38.944 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:28:39.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509455 -lt 64424509456 2024-03-26T00:28:39.277 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:28:40.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:28:40.279 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:28:40.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509457 -lt 64424509456 2024-03-26T00:28:40.600 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:28:40.601 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:28:40.601 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:28:40.975 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-26T00:28:40.975 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:28:40.976 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:28:40.976 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:28:40.976 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:28:40.977 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:28:40.977 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:28:40.977 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:28:41.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-26T00:28:41.260 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:28:41.261 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:28:41.261 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:28:41.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-26T00:28:41.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:28:41.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:28:41.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-26T00:28:41.628 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:931: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ0 2024-03-26T00:28:41.628 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-26T00:28:41.628 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-26T00:28:41.628 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ0 2024-03-26T00:28:41.628 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-26T00:28:41.628 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-26T00:28:41.628 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-26T00:28:42.073 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-26T00:28:42.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-26T00:28:42.735 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-26T00:28:42.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-26T00:28:42.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-26T00:28:42.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ0 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:28:43.119 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: get_osds ecpool MOBJ0 2024-03-26T00:28:43.120 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-26T00:28:43.120 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ0 2024-03-26T00:28:43.121 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ0 2024-03-26T00:28:43.121 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-26T00:28:43.450 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-03-26T00:28:43.450 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:28:43.450 INFO:tasks.workunit.client.0.smithi129.stderr:0' 2024-03-26T00:28:43.450 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-03-26T00:28:43.451 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: osds0=('2' '1' '0') 2024-03-26T00:28:43.451 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: local -a osds0 2024-03-26T00:28:43.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ1 2024-03-26T00:28:43.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-26T00:28:43.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-26T00:28:43.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ1 2024-03-26T00:28:43.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-26T00:28:43.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-26T00:28:43.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-26T00:28:43.889 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is set 2024-03-26T00:28:43.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-26T00:28:44.408 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-26T00:28:44.430 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-26T00:28:44.430 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-26T00:28:44.430 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ1 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:28:44.933 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: get_osds ecpool MOBJ1 2024-03-26T00:28:44.933 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-26T00:28:44.933 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ1 2024-03-26T00:28:44.934 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ1 2024-03-26T00:28:44.934 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-26T00:28:45.289 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-03-26T00:28:45.289 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:28:45.289 INFO:tasks.workunit.client.0.smithi129.stderr:0' 2024-03-26T00:28:45.289 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-03-26T00:28:45.290 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: osds1=('2' '1' '0') 2024-03-26T00:28:45.290 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: local -a osds1 2024-03-26T00:28:45.290 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: seq 0 2 2024-03-26T00:28:45.290 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-26T00:28:45.290 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-26T00:28:45.291 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:28:45.291 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:28:45.291 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:28:45.291 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:28:45.292 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:28:46.603 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:28:46.603 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-26T00:28:46.603 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-26T00:28:46.604 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:28:46.604 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:28:46.605 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:28:46.605 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:28:46.605 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:28:47.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:28:47.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-26T00:28:47.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-26T00:28:47.920 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:28:47.920 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:28:47.921 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:28:47.921 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:28:47.921 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:28:49.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:28:49.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:943: list_missing_erasure_coded: id=2 2024-03-26T00:28:49.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:944: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 MOBJ0 remove 2024-03-26T00:28:51.409 INFO:tasks.workunit.client.0.smithi129.stdout:remove 0#2:a9030505:::MOBJ0:head# 2024-03-26T00:28:51.943 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: list_missing_erasure_coded: id=1 2024-03-26T00:28:51.943 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:947: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ0 remove 2024-03-26T00:28:53.871 INFO:tasks.workunit.client.0.smithi129.stdout:remove 1#2:a9030505:::MOBJ0:head# 2024-03-26T00:28:54.407 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:950: list_missing_erasure_coded: id=1 2024-03-26T00:28:54.407 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:951: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ1 remove 2024-03-26T00:28:55.643 INFO:tasks.workunit.client.0.smithi129.stdout:remove 1#2:8dd16d55:::MOBJ1:head# 2024-03-26T00:28:56.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: id=0 2024-03-26T00:28:56.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:954: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 MOBJ1 remove 2024-03-26T00:28:57.879 INFO:tasks.workunit.client.0.smithi129.stdout:remove 2#2:8dd16d55:::MOBJ1:head# 2024-03-26T00:28:58.412 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: seq 0 2 2024-03-26T00:28:58.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-26T00:28:58.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 0 2024-03-26T00:28:58.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:28:58.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:28:58.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-26T00:28:58.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:28:58.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:28:58.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:28:58.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:28:58.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:28:58.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:28:58.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-26T00:28:58.416 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-26T00:28:58.416 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:28:58.416 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:28:58.416 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:28:58.417 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:28:58.417 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:28:58.417 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:28:58.417 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:28:58.417 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:28:58.417 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:28:58.418 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:28:58.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:28:58.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:28:58.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:28:58.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:28:58.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:28:58.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:28:58.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:28:58.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:28:58.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:28:58.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:28:58.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-26T00:28:58.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-26T00:28:58.420 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.0 2024-03-26T00:28:58.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:28:58.420 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-26T00:28:58.422 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-26T00:28:58.422 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:28:58.423 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:28:58.423 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:28:58.458 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:28:58.456+0000 7fb11acd17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:28:58.461 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:28:58.460+0000 7fb11acd17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:28:58.462 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:28:58.460+0000 7fb11acd17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:28:58.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-26T00:28:58.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:28:58.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-26T00:28:58.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:28:58.744 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:28:58.744 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:28:58.744 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:28:58.744 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:28:58.744 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:28:58.744 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:28:58.995 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:28:58.992+0000 7fb11acd17c0 -1 Falling back to public interface 2024-03-26T00:28:59.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:29:00.063 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:29:00.064 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:29:00.064 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:29:00.064 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:29:00.064 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:29:00.064 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:29:00.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:29:01.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:29:01.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:29:01.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:29:01.390 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:29:01.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:29:01.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:29:01.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:29:01.958 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:29:01.956+0000 7fb11acd17c0 -1 osd.0 35 log_to_monitors true 2024-03-26T00:29:02.715 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:29:02.715 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:29:02.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:29:02.716 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:29:02.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:29:02.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:29:03.087 INFO:tasks.workunit.client.0.smithi129.stderr:osd.0 up in weight 1 up_from 43 up_thru 43 down_at 36 last_clean_interval [5,35) [v2:127.0.0.1:6802/237333623,v1:127.0.0.1:6803/237333623] [v2:127.0.0.1:6804/237333623,v1:127.0.0.1:6805/237333623] exists,up 7a41ffd6-b6c2-499c-9dd7-7d55c626357c 2024-03-26T00:29:03.087 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:29:03.087 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:29:03.087 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:29:03.087 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-26T00:29:03.087 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 1 2024-03-26T00:29:03.088 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:29:03.088 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:29:03.088 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-26T00:29:03.088 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:29:03.088 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:29:03.088 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:29:03.088 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:29:03.088 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:29:03.089 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:29:03.089 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:29:03.089 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:29:03.089 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:29:03.089 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:29:03.089 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:29:03.092 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:29:03.092 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:29:03.092 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:29:03.092 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:29:03.093 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:29:03.093 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:29:03.093 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:29:03.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:29:03.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:29:03.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:29:03.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:29:03.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:29:03.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:29:03.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:29:03.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:29:03.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:29:03.094 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:29:03.094 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:29:03.094 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-26T00:29:03.094 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.1 2024-03-26T00:29:03.094 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:29:03.094 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-26T00:29:03.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-26T00:29:03.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:29:03.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:29:03.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:29:03.118 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:29:03.116+0000 7f7c863087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:29:03.155 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:29:03.152+0000 7f7c863087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:29:03.157 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:29:03.156+0000 7f7c863087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:29:03.450 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-26T00:29:03.450 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:29:03.450 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:29:03.451 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:29:03.451 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:29:03.451 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:29:03.451 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:29:03.451 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:29:03.451 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:29:03.451 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:29:03.691 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:29:03.688+0000 7f7c863087c0 -1 Falling back to public interface 2024-03-26T00:29:03.804 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:29:04.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:29:04.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:29:04.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:29:04.806 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:29:04.807 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:29:04.807 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:29:05.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:29:06.126 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:29:06.126 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:29:06.126 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:29:06.126 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:29:06.127 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:29:06.127 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:29:06.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:29:06.583 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:29:06.564+0000 7f7c863087c0 -1 osd.1 38 log_to_monitors true 2024-03-26T00:29:07.437 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:29:07.438 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:29:07.438 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:29:07.438 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:29:07.438 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:29:07.438 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:29:07.770 INFO:tasks.workunit.client.0.smithi129.stderr:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 39 last_clean_interval [10,38) [v2:127.0.0.1:6810/390671516,v1:127.0.0.1:6811/390671516] [v2:127.0.0.1:6812/390671516,v1:127.0.0.1:6813/390671516] exists,up 736d8177-73db-4bbb-b16e-6271ae1568e6 2024-03-26T00:29:07.770 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:29:07.771 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:29:07.771 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:29:07.771 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-26T00:29:07.771 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 2 2024-03-26T00:29:07.771 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:29:07.771 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:29:07.771 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-26T00:29:07.771 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:29:07.771 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-26T00:29:07.771 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:29:07.771 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:29:07.771 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:29:07.771 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:29:07.771 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-26T00:29:07.771 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-26T00:29:07.772 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:29:07.772 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:29:07.772 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:29:07.773 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:29:07.774 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:29:07.774 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:29:07.774 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:29:07.774 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:29:07.774 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:29:07.774 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:29:07.774 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:29:07.774 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:29:07.774 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:29:07.774 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:29:07.774 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:29:07.774 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:29:07.774 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:29:07.775 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:29:07.775 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:29:07.775 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-26T00:29:07.775 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-26T00:29:07.775 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-26T00:29:07.775 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.2 2024-03-26T00:29:07.775 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-26T00:29:07.776 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-26T00:29:07.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-26T00:29:07.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:29:07.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:29:07.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:29:07.806 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:29:07.804+0000 7fb3f949a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:29:07.936 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:29:07.932+0000 7fb3f949a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:29:07.938 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:29:07.936+0000 7fb3f949a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:29:08.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-26T00:29:08.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:29:08.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-26T00:29:08.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:29:08.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:29:08.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:29:08.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:29:08.142 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:29:08.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:29:08.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:29:08.476 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:29:08.507 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:29:08.504+0000 7fb3f949a7c0 -1 Falling back to public interface 2024-03-26T00:29:09.478 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:29:09.479 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:29:09.479 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:29:09.479 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:29:09.479 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:29:09.479 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:29:09.799 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:29:10.801 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:29:10.801 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:29:10.801 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:29:10.801 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:29:10.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:29:10.802 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:29:11.111 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:29:11.554 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:29:11.551+0000 7fb3f949a7c0 -1 osd.2 40 log_to_monitors true 2024-03-26T00:29:12.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:29:12.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:29:12.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:29:12.113 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:29:12.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:29:12.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:29:12.633 INFO:tasks.workunit.client.0.smithi129.stderr:osd.2 up in weight 1 up_from 49 up_thru 39 down_at 41 last_clean_interval [15,40) [v2:127.0.0.1:6818/2623917719,v1:127.0.0.1:6819/2623917719] [v2:127.0.0.1:6820/2623917719,v1:127.0.0.1:6821/2623917719] exists,up c48eb426-1303-426c-b06d-e79cefe3c319 2024-03-26T00:29:12.633 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:29:12.633 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:29:12.633 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:29:12.633 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:960: list_missing_erasure_coded: create_rbd_pool 2024-03-26T00:29:12.633 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-26T00:29:13.807 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'rbd' removed 2024-03-26T00:29:13.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-26T00:29:13.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-26T00:29:14.781 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'rbd' created 2024-03-26T00:29:14.801 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-26T00:29:15.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-26T00:29:18.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:961: list_missing_erasure_coded: wait_for_clean 2024-03-26T00:29:18.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:29:18.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:29:18.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:29:18.330 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:29:18.331 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:29:18.331 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:29:18.332 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:29:18.332 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:29:18.332 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:29:18.472 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:29:18.472 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:29:18.472 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:29:18.473 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:29:18.473 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:29:18.473 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:29:18.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:29:18.832 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:29:18.832 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-26T00:29:18.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:29:18.833 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:29:18.833 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:29:18.982 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=184683593733 2024-03-26T00:29:18.982 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 184683593733 2024-03-26T00:29:18.982 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-184683593733' 2024-03-26T00:29:18.982 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:29:18.982 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:29:19.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=197568495620 2024-03-26T00:29:19.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 197568495620 2024-03-26T00:29:19.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-184683593733 1-197568495620' 2024-03-26T00:29:19.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:29:19.132 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:29:19.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397507 2024-03-26T00:29:19.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397507 2024-03-26T00:29:19.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-184683593733 1-197568495620 2-210453397507' 2024-03-26T00:29:19.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:29:19.276 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-184683593733 2024-03-26T00:29:19.277 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:29:19.277 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:29:19.278 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:29:19.279 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-184683593733 2024-03-26T00:29:19.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=184683593733 2024-03-26T00:29:19.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 184683593733' 2024-03-26T00:29:19.280 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 184683593733 2024-03-26T00:29:19.280 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:29:19.589 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 184683593732 -lt 184683593733 2024-03-26T00:29:19.589 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:29:20.591 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:29:20.592 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:29:20.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 184683593733 -lt 184683593733 2024-03-26T00:29:20.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:29:20.921 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-197568495620 2024-03-26T00:29:20.921 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:29:20.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:29:20.924 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-197568495620 2024-03-26T00:29:20.925 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:29:20.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=197568495620 2024-03-26T00:29:20.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 197568495620' 2024-03-26T00:29:20.926 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 197568495620 2024-03-26T00:29:20.926 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:29:21.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 197568495620 -lt 197568495620 2024-03-26T00:29:21.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:29:21.282 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-210453397507 2024-03-26T00:29:21.282 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:29:21.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:29:21.286 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-210453397507 2024-03-26T00:29:21.286 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:29:21.287 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397507 2024-03-26T00:29:21.287 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 210453397507 2024-03-26T00:29:21.287 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 210453397507' 2024-03-26T00:29:21.288 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:29:21.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397507 -lt 210453397507 2024-03-26T00:29:21.631 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:29:21.632 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:29:21.632 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:29:22.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-26T00:29:22.032 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:29:22.033 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:29:22.033 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:29:22.033 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:29:22.033 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:29:22.034 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:29:22.034 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:29:22.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-26T00:29:22.330 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:29:22.331 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:29:22.331 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:29:22.725 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-26T00:29:22.725 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:29:22.725 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:29:22.726 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: get_pg ecpool MOBJ0 2024-03-26T00:29:22.726 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-26T00:29:22.726 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=MOBJ0 2024-03-26T00:29:22.726 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool MOBJ0 2024-03-26T00:29:22.727 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-26T00:29:23.066 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: local pg=2.0 2024-03-26T00:29:23.066 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:968: list_missing_erasure_coded: repair 2.0 2024-03-26T00:29:23.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-26T00:29:23.067 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-26T00:29:23.067 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-26T00:29:23.067 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:29:23.067 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:29:23.068 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-26T00:29:23.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-26T00:28:35.029041+0000 2024-03-26T00:29:23.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-26T00:29:23.651 INFO:tasks.workunit.client.0.smithi129.stderr:instructing pg 2.0s0 on osd.2 to repair 2024-03-26T00:29:23.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-26T00:28:35.029041+0000 2024-03-26T00:29:23.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-26T00:29:23.667 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-26T00:28:35.029041+0000 2024-03-26T00:29:23.668 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-26T00:29:23.668 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-26T00:29:23.668 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:29:23.668 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-26T00:29:23.668 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-26T00:29:23.669 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:29:23.669 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:29:23.669 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-26T00:29:24.024 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:28:35.029041+0000 '>' 2024-03-26T00:28:35.029041+0000 2024-03-26T00:29:24.024 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:29:25.026 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:29:25.026 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:29:25.027 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-26T00:29:25.027 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-26T00:29:25.027 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:29:25.028 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:29:25.028 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-26T00:29:25.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:28:35.029041+0000 '>' 2024-03-26T00:28:35.029041+0000 2024-03-26T00:29:25.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:29:26.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:29:26.346 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:29:26.347 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-26T00:29:26.347 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-26T00:29:26.347 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:29:26.348 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:29:26.348 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-26T00:29:26.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:28:35.029041+0000 '>' 2024-03-26T00:28:35.029041+0000 2024-03-26T00:29:26.664 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:29:27.666 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:29:27.666 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:29:27.667 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-26T00:29:27.667 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-26T00:29:27.667 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:29:27.667 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:29:27.668 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-26T00:29:27.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:28:35.029041+0000 '>' 2024-03-26T00:28:35.029041+0000 2024-03-26T00:29:27.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:29:28.969 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:29:28.969 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:29:28.970 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-26T00:29:28.970 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-26T00:29:28.970 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:29:28.971 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:29:28.971 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-26T00:29:29.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:29:24.399375+0000 '>' 2024-03-26T00:28:35.029041+0000 2024-03-26T00:29:29.282 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-26T00:29:29.282 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: seq 0 120 2024-03-26T00:29:29.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: for i in $(seq 0 120) 2024-03-26T00:29:29.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:971: list_missing_erasure_coded: '[' 0 -lt 60 ']' 2024-03-26T00:29:29.285 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: ceph pg 2.0 list_unfound 2024-03-26T00:29:29.285 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: egrep 'MOBJ0|MOBJ1' 2024-03-26T00:29:29.285 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: wc -l 2024-03-26T00:29:29.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: matches=2 2024-03-26T00:29:29.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: '[' 2 -eq 2 ']' 2024-03-26T00:29:29.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: break 2024-03-26T00:29:29.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-26T00:29:29.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-26T00:29:29.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-26T00:29:29.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-26T00:29:29.434 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:29:29.434 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:29:29.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:29:29.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:29:29.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:29:29.543 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:29:29.543 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-26T00:29:29.544 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-26T00:29:29.545 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-26T00:29:29.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-26T00:29:29.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-26T00:29:29.546 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-26T00:29:29.547 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:29:29.547 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-26T00:29:29.547 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-26T00:29:29.548 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:29:29.548 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-26T00:29:29.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-26T00:29:29.550 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-26T00:29:29.570 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-26T00:29:29.570 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:29:29.570 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:29:29.570 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-26T00:29:29.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-26T00:29:29.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-26T00:29:29.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-26T00:29:29.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-26T00:29:29.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-26T00:29:29.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-26T00:29:29.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-26T00:29:29.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-26T00:29:29.572 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-26T00:29:29.573 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:29:29.573 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:29:29.573 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:29:29.573 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:29:29.573 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:29:29.574 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:29:29.574 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-26T00:29:29.575 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-26T00:29:29.576 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-26T00:29:29.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-26T00:29:29.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-26T00:29:29.577 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-26T00:29:29.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:29:29.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-26T00:29:29.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-26T00:29:29.579 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:29:29.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-26T00:29:29.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-26T00:29:29.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-26T00:29:29.582 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-26T00:29:29.582 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:29:29.582 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:29:29.582 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-26T00:29:29.583 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-26T00:29:29.583 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-26T00:29:29.583 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-26T00:29:29.584 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-26T00:29:29.584 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:29:29.584 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:29:29.584 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61636 2024-03-26T00:29:29.585 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-26T00:29:29.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-26T00:29:29.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-26T00:29:29.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-26T00:29:29.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_periodic_scrub_replicated td/osd-scrub-repair 2024-03-26T00:29:29.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5759: TEST_periodic_scrub_replicated: local dir=td/osd-scrub-repair 2024-03-26T00:29:29.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5760: TEST_periodic_scrub_replicated: local poolname=psr_pool 2024-03-26T00:29:29.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5761: TEST_periodic_scrub_replicated: local objname=POBJ 2024-03-26T00:29:29.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5763: TEST_periodic_scrub_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-03-26T00:29:29.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-26T00:29:29.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-26T00:29:29.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-26T00:29:29.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-26T00:29:29.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-26T00:29:29.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-03-26T00:29:30.107 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-26T00:29:30.107 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:29:30.107 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:29:30.108 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:29:30.108 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:29:30.108 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:29:30.108 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:29:30.109 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-03-26T00:29:30.351 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-26T00:29:30.353 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-26T00:29:30.353 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-26T00:29:30.353 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-26T00:29:30.353 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-26T00:29:30.354 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-26T00:29:30.354 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-26T00:29:30.354 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-26T00:29:30.354 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-26T00:29:30.355 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-26T00:29:30.355 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:29:30.355 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:29:30.355 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-26T00:29:30.355 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-26T00:29:30.355 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get fsid 2024-03-26T00:29:30.467 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-26T00:29:30.467 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-26T00:29:30.467 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-26T00:29:30.467 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-26T00:29:30.468 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-26T00:29:30.468 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-26T00:29:30.468 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-26T00:29:30.468 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-26T00:29:30.469 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-26T00:29:30.469 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:29:30.469 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:29:30.469 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-26T00:29:30.470 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-26T00:29:30.470 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get mon_host 2024-03-26T00:29:30.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5764: TEST_periodic_scrub_replicated: run_mgr td/osd-scrub-repair x 2024-03-26T00:29:30.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-26T00:29:30.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-26T00:29:30.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-26T00:29:30.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-26T00:29:30.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-26T00:29:30.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-26T00:29:31.461 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-26T00:29:31.461 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:29:31.461 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:29:31.461 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:29:31.461 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:29:31.461 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:29:31.462 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:29:31.462 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-26T00:29:31.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-26T00:29:31.506 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5765: TEST_periodic_scrub_replicated: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 ' 2024-03-26T00:29:31.506 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5766: TEST_periodic_scrub_replicated: ceph_osd_args+=--osd_scrub_backoff_ratio=0 2024-03-26T00:29:31.506 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5767: TEST_periodic_scrub_replicated: run_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-03-26T00:29:31.506 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-26T00:29:31.506 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-26T00:29:31.506 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-26T00:29:31.506 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-26T00:29:31.506 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:29:31.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:29:31.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:29:31.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:29:31.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:29:31.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-26T00:29:31.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-26T00:29:31.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-26T00:29:31.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-26T00:29:31.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:29:31.507 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-26T00:29:31.507 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:29:31.507 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:29:31.507 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:29:31.508 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:29:31.508 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:29:31.509 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:29:31.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:29:31.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:29:31.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-26T00:29:31.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-26T00:29:31.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:29:31.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:29:31.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:29:31.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:29:31.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:29:31.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:29:31.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-26T00:29:31.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-03-26T00:29:31.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-26T00:29:31.510 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-26T00:29:31.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=528f8cbf-eb3c-4bdb-bfe5-cc565670123d 2024-03-26T00:29:31.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 528f8cbf-eb3c-4bdb-bfe5-cc565670123d' 2024-03-26T00:29:31.511 INFO:tasks.workunit.client.0.smithi129.stdout:add osd0 528f8cbf-eb3c-4bdb-bfe5-cc565670123d 2024-03-26T00:29:31.511 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-26T00:29:31.533 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDrFgJmUXjCHxAAcIcJycYAkeoJMqPISaY9ow== 2024-03-26T00:29:31.533 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDrFgJmUXjCHxAAcIcJycYAkeoJMqPISaY9ow=="}' 2024-03-26T00:29:31.533 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 528f8cbf-eb3c-4bdb-bfe5-cc565670123d -i td/osd-scrub-repair/0/new.json 2024-03-26T00:29:31.821 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-26T00:29:31.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-26T00:29:31.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --mkfs --key AQDrFgJmUXjCHxAAcIcJycYAkeoJMqPISaY9ow== --osd-uuid 528f8cbf-eb3c-4bdb-bfe5-cc565670123d 2024-03-26T00:29:31.873 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:29:31.871+0000 7ffacd9667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:29:31.906 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:29:31.903+0000 7ffacd9667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:29:31.907 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:29:31.903+0000 7ffacd9667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:29:31.907 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:29:31.903+0000 7ffacd9667c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-26T00:29:31.908 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:29:31.903+0000 7ffacd9667c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-26T00:29:31.908 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:29:31.903+0000 7ffacd9667c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-26T00:29:32.008 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:29:32.003+0000 7ffacd9667c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-26T00:29:39.996 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-26T00:29:39.996 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-26T00:29:39.997 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd0 key to auth repository 2024-03-26T00:29:39.997 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-26T00:29:39.997 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-26T00:29:40.374 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.0 2024-03-26T00:29:40.375 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-26T00:29:40.375 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-26T00:29:40.375 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-03-26T00:29:40.375 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-26T00:29:40.375 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-26T00:29:40.408 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:29:40.403+0000 7f1553a5c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:29:40.456 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:29:40.451+0000 7f1553a5c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:29:40.457 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:29:40.455+0000 7f1553a5c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:29:40.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-26T00:29:40.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:29:40.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-26T00:29:40.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:29:40.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:29:40.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:29:40.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:29:40.661 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-26T00:29:40.662 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:29:40.662 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:29:40.958 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:29:41.024 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:29:41.019+0000 7f1553a5c7c0 -1 Falling back to public interface 2024-03-26T00:29:41.960 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:29:41.960 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:29:41.961 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-26T00:29:41.961 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:29:41.961 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:29:41.961 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:29:42.264 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:29:43.266 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:29:43.266 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:29:43.266 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-26T00:29:43.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:29:43.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:29:43.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:29:43.589 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:29:43.920 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:29:43.915+0000 7f1553a5c7c0 -1 osd.0 0 log_to_monitors true 2024-03-26T00:29:44.591 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:29:44.591 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:29:44.591 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-26T00:29:44.592 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:29:44.592 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:29:44.592 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:29:44.989 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:29:45.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:29:45.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:29:45.991 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-26T00:29:45.992 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-26T00:29:45.992 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:29:45.992 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:29:46.317 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:29:47.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:29:47.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:29:47.319 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-26T00:29:47.320 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-26T00:29:47.320 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:29:47.320 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:29:47.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:29:48.745 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:29:48.746 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:29:48.746 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-26T00:29:48.747 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-26T00:29:48.747 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:29:48.747 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:29:49.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:29:50.269 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:29:50.270 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:29:50.270 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-26T00:29:50.270 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-26T00:29:50.271 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:29:50.271 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:29:50.589 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:29:51.591 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:29:51.591 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:29:51.591 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-26T00:29:51.592 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-26T00:29:51.592 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:29:51.592 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:29:51.906 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:29:52.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:29:52.909 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:29:52.909 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-26T00:29:52.909 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-26T00:29:52.909 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:29:52.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:29:53.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:29:54.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:29:54.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:29:54.230 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-26T00:29:54.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-26T00:29:54.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:29:54.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:29:54.551 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:29:55.553 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:29:55.554 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:29:55.554 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-26T00:29:55.554 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-26T00:29:55.555 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:29:55.555 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:29:56.054 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:29:57.056 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:29:57.056 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:29:57.057 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-26T00:29:57.057 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-26T00:29:57.057 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:29:57.057 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:29:57.364 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:29:58.366 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:29:58.367 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:29:58.367 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-26T00:29:58.367 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-26T00:29:58.367 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:29:58.367 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:29:58.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:29:59.678 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:29:59.678 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:29:59.678 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-26T00:29:59.679 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-26T00:29:59.679 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:29:59.679 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:30:00.219 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:30:01.221 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:30:01.221 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:30:01.222 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-26T00:30:01.222 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-26T00:30:01.222 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:30:01.222 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:30:01.539 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:30:02.540 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:30:02.540 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:30:02.541 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-26T00:30:02.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-26T00:30:02.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:30:02.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:30:03.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:30:04.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:30:04.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:30:04.344 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-26T00:30:04.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-26T00:30:04.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:30:04.345 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:30:05.380 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:30:06.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:30:06.382 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:30:06.383 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-26T00:30:06.383 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-26T00:30:06.383 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:30:06.383 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:30:07.428 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:30:08.430 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:30:08.430 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:30:08.430 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-26T00:30:08.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-26T00:30:08.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:30:08.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:30:08.745 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:30:09.747 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:30:09.747 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:30:09.748 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-26T00:30:09.748 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-26T00:30:09.748 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:30:09.748 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:30:10.544 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:30:11.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:30:11.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:30:11.547 INFO:tasks.workunit.client.0.smithi129.stdout:21 2024-03-26T00:30:11.547 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-26T00:30:11.547 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:30:11.547 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:30:12.320 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:30:13.322 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:30:13.323 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:30:13.323 INFO:tasks.workunit.client.0.smithi129.stdout:22 2024-03-26T00:30:13.324 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-26T00:30:13.324 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:30:13.324 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:30:13.837 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:30:14.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:30:14.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:30:14.840 INFO:tasks.workunit.client.0.smithi129.stdout:23 2024-03-26T00:30:14.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-03-26T00:30:14.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:30:14.841 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:30:15.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:30:16.301 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:30:16.301 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:30:16.301 INFO:tasks.workunit.client.0.smithi129.stdout:24 2024-03-26T00:30:16.302 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-03-26T00:30:16.302 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:30:16.302 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:30:16.630 INFO:tasks.workunit.client.0.smithi129.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3913426373,v1:127.0.0.1:6803/3913426373] [v2:127.0.0.1:6804/3913426373,v1:127.0.0.1:6805/3913426373] exists,up 528f8cbf-eb3c-4bdb-bfe5-cc565670123d 2024-03-26T00:30:16.633 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:30:16.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:30:16.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:30:16.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5768: TEST_periodic_scrub_replicated: run_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-03-26T00:30:16.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-26T00:30:16.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-26T00:30:16.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-26T00:30:16.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-26T00:30:16.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:30:16.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:30:16.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:30:16.634 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:30:16.636 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:30:16.636 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:30:16.636 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:30:16.636 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-26T00:30:16.636 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-26T00:30:16.636 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:30:16.636 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-26T00:30:16.636 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:30:16.636 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:30:16.636 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:30:16.637 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:30:16.637 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:30:16.637 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:30:16.637 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:30:16.637 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:30:16.637 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-26T00:30:16.637 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-26T00:30:16.637 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:30:16.637 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:30:16.637 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:30:16.637 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:30:16.637 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:30:16.637 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:30:16.637 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-26T00:30:16.637 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-03-26T00:30:16.637 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:30:16.638 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-26T00:30:16.638 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=aa19053a-4366-4e4c-8de8-04d06bd4db31 2024-03-26T00:30:16.638 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 aa19053a-4366-4e4c-8de8-04d06bd4db31' 2024-03-26T00:30:16.639 INFO:tasks.workunit.client.0.smithi129.stdout:add osd1 aa19053a-4366-4e4c-8de8-04d06bd4db31 2024-03-26T00:30:16.639 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-26T00:30:16.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAYFwJmpo32JhAAvNdm7eap4TLEn9wC1e8KbA== 2024-03-26T00:30:16.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAYFwJmpo32JhAAvNdm7eap4TLEn9wC1e8KbA=="}' 2024-03-26T00:30:16.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new aa19053a-4366-4e4c-8de8-04d06bd4db31 -i td/osd-scrub-repair/1/new.json 2024-03-26T00:30:17.114 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-26T00:30:17.134 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-26T00:30:17.135 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --mkfs --key AQAYFwJmpo32JhAAvNdm7eap4TLEn9wC1e8KbA== --osd-uuid aa19053a-4366-4e4c-8de8-04d06bd4db31 2024-03-26T00:30:17.173 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:30:17.166+0000 7fdd5455b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:30:17.250 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:30:17.246+0000 7fdd5455b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:30:17.251 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:30:17.246+0000 7fdd5455b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:30:17.252 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:30:17.246+0000 7fdd5455b7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-26T00:30:17.252 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:30:17.246+0000 7fdd5455b7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-26T00:30:17.252 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:30:17.246+0000 7fdd5455b7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-26T00:30:17.335 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:30:17.334+0000 7fdd5455b7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-26T00:30:25.395 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-26T00:30:25.396 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-26T00:30:25.397 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-26T00:30:25.397 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd1 key to auth repository 2024-03-26T00:30:25.398 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-26T00:30:25.775 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-26T00:30:25.776 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.1 2024-03-26T00:30:25.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-03-26T00:30:25.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-26T00:30:25.776 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-26T00:30:25.777 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-26T00:30:25.809 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:30:25.806+0000 7fa9539fa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:30:25.838 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:30:25.834+0000 7fa9539fa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:30:25.839 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:30:25.834+0000 7fa9539fa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:30:26.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-26T00:30:26.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:30:26.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:30:26.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:30:26.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:30:26.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:30:26.113 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-26T00:30:26.114 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:30:26.114 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:30:26.114 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:30:26.423 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:30:26.418+0000 7fa9539fa7c0 -1 Falling back to public interface 2024-03-26T00:30:26.432 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:30:27.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:30:27.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:30:27.435 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-26T00:30:27.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:30:27.436 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:30:27.436 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:30:27.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:30:28.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:30:28.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:30:28.757 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-26T00:30:28.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:30:28.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:30:28.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:30:29.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:30:29.487 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:30:29.486+0000 7fa9539fa7c0 -1 osd.1 0 log_to_monitors true 2024-03-26T00:30:30.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:30:30.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:30:30.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:30:30.070 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-26T00:30:30.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:30:30.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:30:30.622 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:30:31.624 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:30:31.624 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:30:31.624 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-26T00:30:31.625 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-26T00:30:31.625 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:30:31.625 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:30:31.974 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:30:32.976 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:30:32.976 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:30:32.976 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-26T00:30:32.977 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-26T00:30:32.977 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:30:32.977 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:30:33.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:30:34.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:30:34.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:30:34.299 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-26T00:30:34.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-26T00:30:34.300 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:30:34.300 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:30:34.814 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:30:35.816 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:30:35.816 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:30:35.816 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-26T00:30:35.817 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-26T00:30:35.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:30:35.817 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:30:36.118 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:30:37.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:30:37.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:30:37.120 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-26T00:30:37.120 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-26T00:30:37.120 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:30:37.120 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:30:37.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:30:38.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:30:38.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:30:38.459 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-26T00:30:38.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-26T00:30:38.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:30:38.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:30:38.954 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:30:39.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:30:39.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:30:39.956 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-26T00:30:39.957 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-26T00:30:39.957 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:30:39.957 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:30:40.278 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:30:41.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:30:41.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:30:41.281 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-26T00:30:41.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-26T00:30:41.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:30:41.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:30:41.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:30:42.741 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:30:42.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:30:42.742 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-26T00:30:42.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-26T00:30:42.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:30:42.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:30:43.098 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:30:44.096 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:30:44.096 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:30:44.097 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-26T00:30:44.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-26T00:30:44.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:30:44.097 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:30:44.439 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:30:45.441 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:30:45.441 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:30:45.442 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-26T00:30:45.443 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-26T00:30:45.443 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:30:45.443 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:30:45.878 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:30:46.880 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:30:46.880 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:30:46.881 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-26T00:30:46.881 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-26T00:30:46.881 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:30:46.881 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:30:47.317 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:30:48.318 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:30:48.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:30:48.319 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-26T00:30:48.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-26T00:30:48.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:30:48.320 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:30:48.754 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:30:49.756 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:30:49.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:30:49.757 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-26T00:30:49.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-26T00:30:49.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:30:49.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:30:50.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:30:51.080 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:30:51.080 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:30:51.080 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-26T00:30:51.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-26T00:30:51.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:30:51.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:30:52.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:30:53.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:30:53.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:30:53.142 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-26T00:30:53.143 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-26T00:30:53.143 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:30:53.143 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:30:53.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:30:54.614 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:30:54.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:30:54.615 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-26T00:30:54.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-26T00:30:54.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:30:54.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:30:55.447 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:30:56.450 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:30:56.450 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:30:56.450 INFO:tasks.workunit.client.0.smithi129.stdout:21 2024-03-26T00:30:56.451 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-26T00:30:56.451 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:30:56.451 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:30:57.161 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:30:58.164 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:30:58.164 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:30:58.164 INFO:tasks.workunit.client.0.smithi129.stdout:22 2024-03-26T00:30:58.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-26T00:30:58.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:30:58.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:30:59.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:31:00.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:31:00.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:31:00.189 INFO:tasks.workunit.client.0.smithi129.stdout:23 2024-03-26T00:31:00.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-03-26T00:31:00.190 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:31:00.190 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:31:00.965 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:31:01.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:31:01.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:31:01.967 INFO:tasks.workunit.client.0.smithi129.stdout:24 2024-03-26T00:31:01.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-03-26T00:31:01.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:31:01.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:31:02.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:31:03.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:31:03.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:31:03.609 INFO:tasks.workunit.client.0.smithi129.stdout:25 2024-03-26T00:31:03.610 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-03-26T00:31:03.610 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:31:03.610 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:31:03.917 INFO:tasks.workunit.client.0.smithi129.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1147115657,v1:127.0.0.1:6811/1147115657] [v2:127.0.0.1:6812/1147115657,v1:127.0.0.1:6813/1147115657] exists,up aa19053a-4366-4e4c-8de8-04d06bd4db31 2024-03-26T00:31:03.918 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:31:03.918 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:31:03.918 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:31:03.918 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5769: TEST_periodic_scrub_replicated: create_rbd_pool 2024-03-26T00:31:03.918 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-26T00:31:04.191 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'rbd' does not exist 2024-03-26T00:31:04.208 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-26T00:31:04.208 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-26T00:31:04.787 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'rbd' created 2024-03-26T00:31:04.807 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-26T00:31:05.809 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-26T00:31:08.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5770: TEST_periodic_scrub_replicated: wait_for_clean 2024-03-26T00:31:08.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:31:08.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:31:08.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:31:08.176 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:31:08.178 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:31:08.178 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:31:08.178 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:31:08.178 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:31:08.178 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:31:08.314 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:31:08.314 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:31:08.314 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:31:08.314 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:31:08.314 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:31:08.315 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:31:08.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:31:08.661 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:31:08.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:31:08.661 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:31:08.661 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:31:08.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836501 2024-03-26T00:31:08.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836501 2024-03-26T00:31:08.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501' 2024-03-26T00:31:08.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:31:08.874 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:31:09.019 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672968 2024-03-26T00:31:09.019 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672968 2024-03-26T00:31:09.019 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-42949672968' 2024-03-26T00:31:09.019 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:31:09.019 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836501 2024-03-26T00:31:09.020 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:31:09.021 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:31:09.022 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836501 2024-03-26T00:31:09.022 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:31:09.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836501 2024-03-26T00:31:09.023 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 21474836501 2024-03-26T00:31:09.023 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-03-26T00:31:09.024 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:31:09.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836501 2024-03-26T00:31:09.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:31:10.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:31:10.333 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:31:10.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836501 2024-03-26T00:31:10.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:31:10.654 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672968 2024-03-26T00:31:10.654 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:31:10.655 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:31:10.657 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672968 2024-03-26T00:31:10.657 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:31:10.658 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672968 2024-03-26T00:31:10.659 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 42949672968 2024-03-26T00:31:10.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2024-03-26T00:31:10.659 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:31:10.995 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672969 -lt 42949672968 2024-03-26T00:31:10.997 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:31:10.997 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:31:10.997 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:31:11.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-26T00:31:11.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:31:11.371 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:31:11.372 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:31:11.372 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:31:11.372 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:31:11.372 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:31:11.372 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:31:11.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-26T00:31:11.670 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:31:11.671 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:31:11.671 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:31:12.016 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-26T00:31:12.016 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:31:12.016 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:31:12.016 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5772: TEST_periodic_scrub_replicated: create_pool psr_pool 1 1 2024-03-26T00:31:12.016 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool 1 1 2024-03-26T00:31:12.483 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'psr_pool' created 2024-03-26T00:31:12.503 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-26T00:31:13.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5773: TEST_periodic_scrub_replicated: wait_for_clean 2024-03-26T00:31:13.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:31:13.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:31:13.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:31:13.506 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:31:13.507 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:31:13.507 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:31:13.508 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:31:13.508 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:31:13.508 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:31:13.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:31:13.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:31:13.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:31:13.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:31:13.649 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:31:13.650 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:31:14.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:31:14.165 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:31:14.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:31:14.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:31:14.166 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:31:14.316 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836503 2024-03-26T00:31:14.316 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836503 2024-03-26T00:31:14.316 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503' 2024-03-26T00:31:14.316 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:31:14.316 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:31:14.503 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672971 2024-03-26T00:31:14.503 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672971 2024-03-26T00:31:14.503 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672971' 2024-03-26T00:31:14.503 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:31:14.504 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836503 2024-03-26T00:31:14.504 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:31:14.505 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:31:14.506 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836503 2024-03-26T00:31:14.507 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:31:14.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836503 2024-03-26T00:31:14.508 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 21474836503 2024-03-26T00:31:14.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-03-26T00:31:14.509 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:31:14.881 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836503 2024-03-26T00:31:14.882 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:31:15.883 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:31:15.884 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:31:16.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836503 2024-03-26T00:31:16.201 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:31:17.203 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-26T00:31:17.204 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:31:17.506 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836503 2024-03-26T00:31:17.506 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:31:17.506 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672971 2024-03-26T00:31:17.507 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:31:17.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:31:17.509 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672971 2024-03-26T00:31:17.510 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:31:17.512 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672971 2024-03-26T00:31:17.512 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2024-03-26T00:31:17.513 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 42949672971 2024-03-26T00:31:17.513 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:31:17.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672971 -lt 42949672971 2024-03-26T00:31:17.909 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:31:17.909 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:31:17.909 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:31:18.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-26T00:31:18.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:31:18.284 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:31:18.284 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:31:18.284 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:31:18.284 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:31:18.284 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:31:18.284 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:31:18.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-26T00:31:18.599 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:31:18.600 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:31:18.600 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:31:18.959 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-26T00:31:18.960 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:31:18.960 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:31:18.960 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5775: TEST_periodic_scrub_replicated: local osd=0 2024-03-26T00:31:18.960 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5776: TEST_periodic_scrub_replicated: add_something td/osd-scrub-repair psr_pool POBJ scrub 2024-03-26T00:31:18.960 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-26T00:31:18.960 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=psr_pool 2024-03-26T00:31:18.960 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=POBJ 2024-03-26T00:31:18.960 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=scrub 2024-03-26T00:31:18.960 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' scrub = noscrub ']' 2024-03-26T00:31:18.960 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:81: add_something: ceph osd unset noscrub 2024-03-26T00:31:19.437 INFO:tasks.workunit.client.0.smithi129.stderr:noscrub is unset 2024-03-26T00:31:19.456 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:82: add_something: ceph osd unset nodeep-scrub 2024-03-26T00:31:19.964 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is unset 2024-03-26T00:31:19.982 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-26T00:31:19.982 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-26T00:31:19.982 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool psr_pool put POBJ td/osd-scrub-repair/ORIGINAL 2024-03-26T00:31:20.288 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5777: TEST_periodic_scrub_replicated: get_primary psr_pool POBJ 2024-03-26T00:31:20.288 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=psr_pool 2024-03-26T00:31:20.288 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=POBJ 2024-03-26T00:31:20.288 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map psr_pool POBJ 2024-03-26T00:31:20.289 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-26T00:31:20.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5777: TEST_periodic_scrub_replicated: local primary=0 2024-03-26T00:31:20.609 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5778: TEST_periodic_scrub_replicated: get_pg psr_pool POBJ 2024-03-26T00:31:20.609 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=psr_pool 2024-03-26T00:31:20.609 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=POBJ 2024-03-26T00:31:20.609 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map psr_pool POBJ 2024-03-26T00:31:20.609 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-26T00:31:20.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5778: TEST_periodic_scrub_replicated: local pg=2.0 2024-03-26T00:31:20.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5781: TEST_periodic_scrub_replicated: local payload=UVWXYZ 2024-03-26T00:31:20.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5782: TEST_periodic_scrub_replicated: echo UVWXYZ 2024-03-26T00:31:20.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5784: TEST_periodic_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-03-26T00:31:20.912 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-26T00:31:20.912 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-26T00:31:20.912 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-26T00:31:20.912 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-26T00:31:20.912 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-03-26T00:31:20.912 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-26T00:31:20.912 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-26T00:31:20.912 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-26T00:31:20.912 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-26T00:31:20.912 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-26T00:31:20.912 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:31:20.912 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:31:20.912 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:31:20.912 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:31:20.913 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:31:22.224 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:31:22.224 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-03-26T00:31:22.224 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:31:22.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:31:22.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:31:22.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:31:22.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:31:22.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-03-26T00:31:24.521 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-03-26T00:31:24.521 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:31:24.521 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:31:24.521 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-26T00:31:24.521 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:31:24.521 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:31:24.521 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:31:24.521 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:31:24.521 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:31:24.521 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:31:24.522 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-26T00:31:24.522 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-26T00:31:24.522 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:31:24.522 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:31:24.522 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:31:24.522 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:31:24.522 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:31:24.522 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:31:24.523 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:31:24.524 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:31:24.524 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:31:24.524 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:31:24.524 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:31:24.524 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:31:24.524 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:31:24.524 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:31:24.524 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:31:24.524 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:31:24.524 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:31:24.524 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:31:24.524 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:31:24.524 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-03-26T00:31:24.525 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-26T00:31:24.525 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-26T00:31:24.525 INFO:tasks.workunit.client.0.smithi129.stderr:start osd.0 2024-03-26T00:31:24.525 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-03-26T00:31:24.526 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-26T00:31:24.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-26T00:31:24.527 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:31:24.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:31:24.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:31:24.561 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:31:24.560+0000 7fb5ad2b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:31:24.564 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:31:24.560+0000 7fb5ad2b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:31:24.566 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:31:24.564+0000 7fb5ad2b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:31:24.850 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-26T00:31:24.850 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:31:24.851 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-26T00:31:24.851 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:31:24.851 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:31:24.851 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:31:24.851 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:31:24.851 INFO:tasks.workunit.client.0.smithi129.stderr:0 2024-03-26T00:31:24.851 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:31:24.851 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:31:25.092 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:31:25.088+0000 7fb5ad2b77c0 -1 Falling back to public interface 2024-03-26T00:31:25.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:31:26.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:31:26.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:31:26.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:31:26.168 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:31:26.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:31:26.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:31:26.483 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:31:27.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:31:27.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:31:27.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:31:27.485 INFO:tasks.workunit.client.0.smithi129.stderr:2 2024-03-26T00:31:27.486 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:31:27.486 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:31:27.798 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:31:28.028 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:31:28.024+0000 7fb5ad2b77c0 -1 osd.0 23 log_to_monitors true 2024-03-26T00:31:28.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:31:28.800 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:31:28.801 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:31:28.801 INFO:tasks.workunit.client.0.smithi129.stderr:3 2024-03-26T00:31:28.801 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:31:28.801 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:31:29.392 INFO:tasks.workunit.client.0.smithi129.stderr:osd.0 up in weight 1 up_from 27 up_thru 19 down_at 24 last_clean_interval [5,23) [v2:127.0.0.1:6802/1129275575,v1:127.0.0.1:6803/1129275575] [v2:127.0.0.1:6804/1129275575,v1:127.0.0.1:6805/1129275575] exists,up 528f8cbf-eb3c-4bdb-bfe5-cc565670123d 2024-03-26T00:31:29.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:31:29.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:31:29.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:31:29.393 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-26T00:31:29.393 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:31:29.393 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:31:29.393 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:31:29.393 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:31:29.394 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:31:29.394 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:31:29.395 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:31:29.395 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:31:29.395 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:31:29.522 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:31:29.523 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:31:29.523 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:31:29.523 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:31:29.523 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:31:29.523 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:31:29.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:31:29.840 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:31:29.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:31:29.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:31:29.841 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:31:29.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964116994 2024-03-26T00:31:29.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964116994 2024-03-26T00:31:29.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-115964116994' 2024-03-26T00:31:29.987 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:31:29.988 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:31:30.193 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672975 2024-03-26T00:31:30.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672975 2024-03-26T00:31:30.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-115964116994 1-42949672975' 2024-03-26T00:31:30.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:31:30.194 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-115964116994 2024-03-26T00:31:30.195 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:31:30.195 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:31:30.196 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-115964116994 2024-03-26T00:31:30.196 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:31:30.196 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964116994 2024-03-26T00:31:30.196 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 115964116994' 2024-03-26T00:31:30.196 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.0 seq 115964116994 2024-03-26T00:31:30.197 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:31:30.557 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964116994 -lt 115964116994 2024-03-26T00:31:30.557 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:31:30.558 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672975 2024-03-26T00:31:30.558 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:31:30.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:31:30.561 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672975 2024-03-26T00:31:30.561 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:31:30.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672975 2024-03-26T00:31:30.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-03-26T00:31:30.562 INFO:tasks.workunit.client.0.smithi129.stderr:waiting osd.1 seq 42949672975 2024-03-26T00:31:30.563 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:31:30.879 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672974 -lt 42949672975 2024-03-26T00:31:30.880 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:31:31.881 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:31:31.882 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:31:32.193 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672974 -lt 42949672975 2024-03-26T00:31:32.193 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:31:33.195 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-26T00:31:33.196 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:31:33.518 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672975 -lt 42949672975 2024-03-26T00:31:33.518 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:31:33.519 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:31:33.519 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:31:33.939 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-26T00:31:33.939 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:31:33.940 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:31:33.940 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:31:33.940 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:31:33.940 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:31:33.940 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:31:33.940 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:31:34.287 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-26T00:31:34.287 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:31:34.288 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:31:34.288 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:31:34.641 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-26T00:31:34.641 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:31:34.642 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:31:34.642 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5787: TEST_periodic_scrub_replicated: set -o pipefail 2024-03-26T00:31:34.642 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5788: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-03-26T00:31:34.642 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5788: TEST_periodic_scrub_replicated: jq . 2024-03-26T00:31:34.701 INFO:tasks.workunit.client.0.smithi129.stderr:No scrub information available for pg 2.0 2024-03-26T00:31:34.701 INFO:tasks.workunit.client.0.smithi129.stderr:error 2: (2) No such file or directory 2024-03-26T00:31:34.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5789: TEST_periodic_scrub_replicated: set +o pipefail 2024-03-26T00:31:34.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5791: TEST_periodic_scrub_replicated: pg_deep_scrub 2.0 2024-03-26T00:31:34.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=2.0 2024-03-26T00:31:34.704 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-03-26T00:31:34.704 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-26T00:31:34.704 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-26T00:31:34.704 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:31:34.704 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-03-26T00:31:35.001 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-03-26T00:31:12.397607+0000 2024-03-26T00:31:35.001 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 2.0 2024-03-26T00:31:35.283 INFO:tasks.workunit.client.0.smithi129.stderr:instructing pg 2.0 on osd.0 to deep-scrub 2024-03-26T00:31:35.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 2.0 2024-03-26T00:31:12.397607+0000 last_deep_scrub_stamp 2024-03-26T00:31:35.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-26T00:31:35.300 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-26T00:31:12.397607+0000 2024-03-26T00:31:35.300 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-03-26T00:31:35.300 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-26T00:31:35.300 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:31:35.301 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-03-26T00:31:35.301 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-26T00:31:35.301 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-26T00:31:35.301 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:31:35.301 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-03-26T00:31:35.612 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:31:12.397607+0000 '>' 2024-03-26T00:31:12.397607+0000 2024-03-26T00:31:35.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:31:36.614 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:31:36.614 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:31:36.615 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-03-26T00:31:36.615 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-26T00:31:36.615 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-26T00:31:36.616 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:31:36.616 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-03-26T00:31:36.934 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:31:12.397607+0000 '>' 2024-03-26T00:31:12.397607+0000 2024-03-26T00:31:36.934 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:31:37.935 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:31:37.935 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:31:37.937 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-03-26T00:31:37.937 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-26T00:31:37.937 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-26T00:31:37.937 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:31:37.938 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-03-26T00:31:38.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:31:12.397607+0000 '>' 2024-03-26T00:31:12.397607+0000 2024-03-26T00:31:38.261 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:31:39.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:31:39.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:31:39.263 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-03-26T00:31:39.264 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-26T00:31:39.264 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-26T00:31:39.264 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:31:39.264 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-03-26T00:31:39.583 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:31:12.397607+0000 '>' 2024-03-26T00:31:12.397607+0000 2024-03-26T00:31:39.583 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:31:40.584 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:31:40.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:31:40.585 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-03-26T00:31:40.585 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-26T00:31:40.586 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-26T00:31:40.586 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:31:40.586 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-03-26T00:31:40.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:31:35.953183+0000 '>' 2024-03-26T00:31:12.397607+0000 2024-03-26T00:31:40.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-26T00:31:40.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5794: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-03-26T00:31:40.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5794: TEST_periodic_scrub_replicated: grep -q POBJ 2024-03-26T00:31:40.912 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5794: TEST_periodic_scrub_replicated: jq . 2024-03-26T00:31:40.947 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5796: TEST_periodic_scrub_replicated: flush_pg_stats 2024-03-26T00:31:40.947 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:31:40.947 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:31:41.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:31:41.240 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:31:41.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:31:41.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:31:41.241 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:31:41.439 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964116997 2024-03-26T00:31:41.439 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964116997 2024-03-26T00:31:41.440 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-115964116997' 2024-03-26T00:31:41.440 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:31:41.440 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:31:41.576 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672978 2024-03-26T00:31:41.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672978 2024-03-26T00:31:41.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-115964116997 1-42949672978' 2024-03-26T00:31:41.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:31:41.577 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-115964116997 2024-03-26T00:31:41.577 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:31:41.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:31:41.579 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-115964116997 2024-03-26T00:31:41.579 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:31:41.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964116997 2024-03-26T00:31:41.580 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 115964116997 2024-03-26T00:31:41.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 115964116997' 2024-03-26T00:31:41.581 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:31:41.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964116996 -lt 115964116997 2024-03-26T00:31:41.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:31:42.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:31:42.899 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:31:43.205 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964116997 -lt 115964116997 2024-03-26T00:31:43.206 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:31:43.207 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672978 2024-03-26T00:31:43.207 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:31:43.208 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:31:43.209 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672978 2024-03-26T00:31:43.209 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:31:43.211 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672978 2024-03-26T00:31:43.212 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-03-26T00:31:43.212 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 42949672978 2024-03-26T00:31:43.212 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:31:43.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672978 -lt 42949672978 2024-03-26T00:31:43.565 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5797: TEST_periodic_scrub_replicated: get_last_scrub_stamp 2.0 2024-03-26T00:31:43.566 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-26T00:31:43.566 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:31:43.566 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-26T00:31:43.566 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:31:43.855 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5797: TEST_periodic_scrub_replicated: local last_scrub=2024-03-26T00:31:35.953183+0000 2024-03-26T00:31:43.855 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5799: TEST_periodic_scrub_replicated: ceph tell 2.0 schedule-scrub 2024-03-26T00:31:43.989 INFO:tasks.workunit.client.0.smithi129.stdout:{ 2024-03-26T00:31:43.989 INFO:tasks.workunit.client.0.smithi129.stdout: "deep": false, 2024-03-26T00:31:43.989 INFO:tasks.workunit.client.0.smithi129.stdout: "must": false, 2024-03-26T00:31:43.989 INFO:tasks.workunit.client.0.smithi129.stdout: "stamp": "2024-03-19T00:30:03.989549+0000" 2024-03-26T00:31:43.989 INFO:tasks.workunit.client.0.smithi129.stdout:} 2024-03-26T00:31:44.003 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5801: TEST_periodic_scrub_replicated: wait_for_scrub 2.0 2024-03-26T00:31:35.953183+0000 2024-03-26T00:31:44.003 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-26T00:31:44.003 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-26T00:31:35.953183+0000 2024-03-26T00:31:44.003 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-26T00:31:44.003 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-26T00:31:44.003 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:31:44.003 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-26T00:31:44.004 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-26T00:31:44.004 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:31:44.004 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:31:44.004 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-26T00:31:44.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:31:35.953183+0000 '>' 2024-03-26T00:31:35.953183+0000 2024-03-26T00:31:44.309 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:31:45.311 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:31:45.311 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:31:45.312 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-26T00:31:45.312 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-26T00:31:45.312 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:31:45.312 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:31:45.312 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-26T00:31:45.617 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:31:35.953183+0000 '>' 2024-03-26T00:31:35.953183+0000 2024-03-26T00:31:45.617 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:31:46.619 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:31:46.619 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:31:46.620 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-26T00:31:46.620 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-26T00:31:46.620 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:31:46.621 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:31:46.621 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-26T00:31:46.925 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:31:35.953183+0000 '>' 2024-03-26T00:31:35.953183+0000 2024-03-26T00:31:46.926 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:31:47.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:31:47.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:31:47.928 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-26T00:31:47.928 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-26T00:31:47.928 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:31:47.929 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:31:47.929 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-26T00:31:48.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:31:35.953183+0000 '>' 2024-03-26T00:31:35.953183+0000 2024-03-26T00:31:48.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:31:49.233 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:31:49.233 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:31:49.234 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-26T00:31:49.234 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-26T00:31:49.235 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:31:49.235 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:31:49.235 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-26T00:31:49.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:31:35.953183+0000 '>' 2024-03-26T00:31:35.953183+0000 2024-03-26T00:31:49.542 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:31:50.543 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:31:50.543 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:31:50.545 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-26T00:31:50.545 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-26T00:31:50.545 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:31:50.545 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:31:50.546 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-26T00:31:50.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:31:44.960165+0000 '>' 2024-03-26T00:31:35.953183+0000 2024-03-26T00:31:50.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-26T00:31:50.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5804: TEST_periodic_scrub_replicated: grep -q 'Deep scrub errors, upgrading scrub to deep-scrub' td/osd-scrub-repair/osd.0.log 2024-03-26T00:31:50.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5807: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-03-26T00:31:50.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5807: TEST_periodic_scrub_replicated: jq . 2024-03-26T00:31:50.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5807: TEST_periodic_scrub_replicated: grep -q POBJ 2024-03-26T00:31:50.955 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5810: TEST_periodic_scrub_replicated: ceph osd set nodeep-scrub 2024-03-26T00:31:51.367 INFO:tasks.workunit.client.0.smithi129.stderr:nodeep-scrub is set 2024-03-26T00:31:51.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5812: TEST_periodic_scrub_replicated: ceph tell osd.0 get_latest_osdmap 2024-03-26T00:31:51.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5813: TEST_periodic_scrub_replicated: flush_pg_stats 2024-03-26T00:31:51.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:31:51.579 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:31:51.894 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:31:51.894 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:31:51.894 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:31:51.894 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:31:51.895 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:31:52.087 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964117000 2024-03-26T00:31:52.087 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964117000 2024-03-26T00:31:52.087 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-115964117000' 2024-03-26T00:31:52.087 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:31:52.087 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:31:52.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672981 2024-03-26T00:31:52.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672981 2024-03-26T00:31:52.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-115964117000 1-42949672981' 2024-03-26T00:31:52.227 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:31:52.228 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-115964117000 2024-03-26T00:31:52.228 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:31:52.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:31:52.230 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-115964117000 2024-03-26T00:31:52.230 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:31:52.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964117000 2024-03-26T00:31:52.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 115964117000' 2024-03-26T00:31:52.231 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 115964117000 2024-03-26T00:31:52.232 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:31:52.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964117000 -lt 115964117000 2024-03-26T00:31:52.533 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:31:52.533 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672981 2024-03-26T00:31:52.534 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:31:52.536 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:31:52.537 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672981 2024-03-26T00:31:52.537 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:31:52.538 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672981 2024-03-26T00:31:52.539 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 42949672981 2024-03-26T00:31:52.539 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-03-26T00:31:52.539 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:31:52.864 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672980 -lt 42949672981 2024-03-26T00:31:52.865 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:31:53.866 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:31:53.867 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:31:54.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672980 -lt 42949672981 2024-03-26T00:31:54.178 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:31:55.179 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-26T00:31:55.181 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:31:55.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672981 -lt 42949672981 2024-03-26T00:31:55.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5814: TEST_periodic_scrub_replicated: sleep 5 2024-03-26T00:32:00.491 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5817: TEST_periodic_scrub_replicated: ceph tell 2.0 schedule-scrub 2024-03-26T00:32:00.636 INFO:tasks.workunit.client.0.smithi129.stdout:{ 2024-03-26T00:32:00.636 INFO:tasks.workunit.client.0.smithi129.stdout: "deep": false, 2024-03-26T00:32:00.636 INFO:tasks.workunit.client.0.smithi129.stdout: "must": false, 2024-03-26T00:32:00.636 INFO:tasks.workunit.client.0.smithi129.stdout: "stamp": "2024-03-19T00:30:20.636849+0000" 2024-03-26T00:32:00.636 INFO:tasks.workunit.client.0.smithi129.stdout:} 2024-03-26T00:32:00.651 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5820: TEST_periodic_scrub_replicated: local found=false 2024-03-26T00:32:00.651 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5821: TEST_periodic_scrub_replicated: seq 14 -1 0 2024-03-26T00:32:00.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5821: TEST_periodic_scrub_replicated: for i in $(seq 14 -1 0) 2024-03-26T00:32:00.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5823: TEST_periodic_scrub_replicated: sleep 1 2024-03-26T00:32:01.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5824: TEST_periodic_scrub_replicated: grep -q 'Regular scrub skipped due to deep-scrub errors and nodeep-scrub set' td/osd-scrub-repair/osd.0.log 2024-03-26T00:32:01.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5824: TEST_periodic_scrub_replicated: found=true 2024-03-26T00:32:01.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5824: TEST_periodic_scrub_replicated: break 2024-03-26T00:32:01.659 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5827: TEST_periodic_scrub_replicated: test true = true 2024-03-26T00:32:01.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5830: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-03-26T00:32:01.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5830: TEST_periodic_scrub_replicated: jq . 2024-03-26T00:32:01.660 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5830: TEST_periodic_scrub_replicated: grep -q POBJ 2024-03-26T00:32:01.701 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5832: TEST_periodic_scrub_replicated: flush_pg_stats 2024-03-26T00:32:01.701 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:32:01.702 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:32:02.008 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:32:02.008 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:32:02.008 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:32:02.008 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:32:02.009 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:32:02.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964117003 2024-03-26T00:32:02.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964117003 2024-03-26T00:32:02.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-115964117003' 2024-03-26T00:32:02.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:32:02.236 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:32:02.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672984 2024-03-26T00:32:02.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672984 2024-03-26T00:32:02.379 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-115964117003 1-42949672984' 2024-03-26T00:32:02.380 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:32:02.380 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-115964117003 2024-03-26T00:32:02.380 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:32:02.381 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:32:02.382 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-115964117003 2024-03-26T00:32:02.382 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:32:02.384 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964117003 2024-03-26T00:32:02.384 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 115964117003' 2024-03-26T00:32:02.384 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 115964117003 2024-03-26T00:32:02.384 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:32:02.687 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964117002 -lt 115964117003 2024-03-26T00:32:02.687 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:32:03.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:32:03.690 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:32:04.002 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964117002 -lt 115964117003 2024-03-26T00:32:04.002 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:32:05.004 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-26T00:32:05.005 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:32:05.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964117004 -lt 115964117003 2024-03-26T00:32:05.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:32:05.331 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:32:05.331 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672984 2024-03-26T00:32:05.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:32:05.333 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672984 2024-03-26T00:32:05.334 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:32:05.334 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672984 2024-03-26T00:32:05.335 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 42949672984 2024-03-26T00:32:05.335 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-03-26T00:32:05.335 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:32:05.621 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672984 -lt 42949672984 2024-03-26T00:32:05.622 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5834: TEST_periodic_scrub_replicated: pg_schedule_scrub 2.0 2024-03-26T00:32:05.622 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1917: pg_schedule_scrub: local pgid=2.0 2024-03-26T00:32:05.622 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1918: pg_schedule_scrub: get_last_scrub_stamp 2.0 2024-03-26T00:32:05.622 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-26T00:32:05.622 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:32:05.623 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-26T00:32:05.623 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:32:05.976 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1918: pg_schedule_scrub: local last_scrub=2024-03-19T00:30:20.636849+0000 2024-03-26T00:32:05.976 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1919: pg_schedule_scrub: ceph pg scrub 2.0 2024-03-26T00:32:06.290 INFO:tasks.workunit.client.0.smithi129.stderr:instructing pg 2.0 on osd.0 to scrub 2024-03-26T00:32:06.306 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1920: pg_schedule_scrub: wait_for_scrub 2.0 2024-03-19T00:30:20.636849+0000 2024-03-26T00:32:06.306 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-26T00:32:06.306 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-19T00:30:20.636849+0000 2024-03-26T00:32:06.306 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-26T00:32:06.306 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-26T00:32:06.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:32:06.307 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-26T00:32:06.307 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-26T00:32:06.307 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:32:06.307 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-26T00:32:06.308 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:32:06.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-19T00:30:20.636849+0000 '>' 2024-03-19T00:30:20.636849+0000 2024-03-26T00:32:06.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:32:07.597 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:32:07.597 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:32:07.598 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-26T00:32:07.598 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-26T00:32:07.598 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:32:07.599 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:32:07.599 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-26T00:32:07.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-19T00:30:20.636849+0000 '>' 2024-03-19T00:30:20.636849+0000 2024-03-26T00:32:07.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:32:08.913 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:32:08.913 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:32:08.914 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-26T00:32:08.914 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-26T00:32:08.914 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:32:08.915 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:32:08.915 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-26T00:32:09.228 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-19T00:30:20.636849+0000 '>' 2024-03-19T00:30:20.636849+0000 2024-03-26T00:32:09.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:32:10.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:32:10.230 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:32:10.231 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-26T00:32:10.232 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-26T00:32:10.232 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:32:10.232 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:32:10.233 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-26T00:32:10.537 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:32:06.928734+0000 '>' 2024-03-19T00:30:20.636849+0000 2024-03-26T00:32:10.537 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-26T00:32:10.537 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5835: TEST_periodic_scrub_replicated: grep -q 'Regular scrub request, deep-scrub details will be lost' td/osd-scrub-repair/osd.0.log 2024-03-26T00:32:10.540 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5838: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-03-26T00:32:10.540 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5838: TEST_periodic_scrub_replicated: jq . 2024-03-26T00:32:10.540 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5838: TEST_periodic_scrub_replicated: grep -qv POBJ 2024-03-26T00:32:10.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-26T00:32:10.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-26T00:32:10.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-26T00:32:10.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-26T00:32:10.568 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:32:10.569 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:32:10.569 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:32:10.569 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:32:10.569 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:32:10.879 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:32:10.879 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-26T00:32:10.880 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-26T00:32:10.881 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-26T00:32:10.882 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-26T00:32:10.882 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-26T00:32:10.883 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-26T00:32:10.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:32:10.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-26T00:32:10.885 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-26T00:32:10.885 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:32:10.886 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-26T00:32:10.888 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-26T00:32:10.888 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-26T00:32:10.907 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-26T00:32:10.907 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:32:10.907 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:32:10.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-26T00:32:10.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-26T00:32:10.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-26T00:32:10.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-26T00:32:10.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-26T00:32:10.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-26T00:32:10.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-26T00:32:10.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-26T00:32:10.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-26T00:32:10.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-26T00:32:10.911 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:32:10.911 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:32:10.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:32:10.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:32:10.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:32:10.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:32:10.915 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-26T00:32:10.916 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-26T00:32:10.917 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-26T00:32:10.918 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-26T00:32:10.918 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-26T00:32:10.919 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-26T00:32:10.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:32:10.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-26T00:32:10.920 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-26T00:32:10.921 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:32:10.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-26T00:32:10.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-26T00:32:10.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-26T00:32:10.925 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-26T00:32:10.925 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:32:10.925 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:32:10.925 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-26T00:32:10.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-26T00:32:10.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-26T00:32:10.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-26T00:32:10.929 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-26T00:32:10.929 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:32:10.929 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:32:10.929 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61636 2024-03-26T00:32:10.931 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-26T00:32:10.933 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-26T00:32:10.933 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-26T00:32:10.933 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-26T00:32:10.933 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_repair_stats td/osd-scrub-repair 2024-03-26T00:32:10.933 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:654: TEST_repair_stats: local dir=td/osd-scrub-repair 2024-03-26T00:32:10.933 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:655: TEST_repair_stats: local poolname=testpool 2024-03-26T00:32:10.933 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:656: TEST_repair_stats: local OSDS=2 2024-03-26T00:32:10.933 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:657: TEST_repair_stats: local OBJS=30 2024-03-26T00:32:10.933 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:659: TEST_repair_stats: local REPAIRS=20 2024-03-26T00:32:10.933 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:662: TEST_repair_stats: run_mon td/osd-scrub-repair a 2024-03-26T00:32:10.933 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-26T00:32:10.934 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-26T00:32:10.934 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-26T00:32:10.934 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-26T00:32:10.934 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-26T00:32:10.934 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-26T00:32:11.468 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-26T00:32:11.468 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:32:11.468 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:32:11.469 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:32:11.469 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:32:11.469 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:32:11.469 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:32:11.470 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-26T00:32:11.727 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-26T00:32:11.728 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-26T00:32:11.728 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-26T00:32:11.728 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-26T00:32:11.728 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-26T00:32:11.729 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-26T00:32:11.729 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-26T00:32:11.729 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-26T00:32:11.729 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-26T00:32:11.729 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-26T00:32:11.729 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:32:11.729 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:32:11.730 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-26T00:32:11.730 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-26T00:32:11.730 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get fsid 2024-03-26T00:32:11.845 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-26T00:32:11.845 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-26T00:32:11.846 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-26T00:32:11.846 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-26T00:32:11.846 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-26T00:32:11.846 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-26T00:32:11.846 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-26T00:32:11.847 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-26T00:32:11.847 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-26T00:32:11.847 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:32:11.847 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:32:11.847 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-26T00:32:11.848 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-26T00:32:11.848 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get mon_host 2024-03-26T00:32:11.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:663: TEST_repair_stats: run_mgr td/osd-scrub-repair x 2024-03-26T00:32:11.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-26T00:32:11.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-26T00:32:11.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-26T00:32:11.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-26T00:32:11.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-26T00:32:11.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-26T00:32:12.854 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-26T00:32:12.855 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:32:12.855 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:32:12.855 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:32:12.855 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:32:12.855 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:32:12.855 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:32:12.856 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-26T00:32:12.857 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-26T00:32:12.891 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:665: TEST_repair_stats: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-26T00:32:12.891 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: expr 2 - 1 2024-03-26T00:32:12.893 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: seq 0 1 2024-03-26T00:32:12.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: for id in $(seq 0 $(expr $OSDS - 1)) 2024-03-26T00:32:12.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:667: TEST_repair_stats: run_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-26T00:32:12.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-26T00:32:12.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-26T00:32:12.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-26T00:32:12.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-26T00:32:12.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:32:12.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:32:12.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:32:12.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:32:12.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:32:12.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-26T00:32:12.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-26T00:32:12.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-26T00:32:12.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-26T00:32:12.897 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:32:12.897 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-26T00:32:12.897 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:32:12.897 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:32:12.897 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:32:12.897 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:32:12.898 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:32:12.898 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:32:12.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:32:12.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:32:12.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-26T00:32:12.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-26T00:32:12.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:32:12.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:32:12.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:32:12.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:32:12.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:32:12.898 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:32:12.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-26T00:32:12.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-26T00:32:12.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-26T00:32:12.899 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-26T00:32:12.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ebf71738-9d30-40f8-aded-e8cf86f55f5e 2024-03-26T00:32:12.900 INFO:tasks.workunit.client.0.smithi129.stdout:add osd0 ebf71738-9d30-40f8-aded-e8cf86f55f5e 2024-03-26T00:32:12.900 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ebf71738-9d30-40f8-aded-e8cf86f55f5e' 2024-03-26T00:32:12.900 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-26T00:32:12.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCMFwJmGYykNhAAZVR5AaNqLD8mug6wngV4Gg== 2024-03-26T00:32:12.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCMFwJmGYykNhAAZVR5AaNqLD8mug6wngV4Gg=="}' 2024-03-26T00:32:12.917 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ebf71738-9d30-40f8-aded-e8cf86f55f5e -i td/osd-scrub-repair/0/new.json 2024-03-26T00:32:13.232 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-26T00:32:13.248 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-26T00:32:13.249 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCMFwJmGYykNhAAZVR5AaNqLD8mug6wngV4Gg== --osd-uuid ebf71738-9d30-40f8-aded-e8cf86f55f5e 2024-03-26T00:32:13.284 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:32:13.279+0000 7f6f70db27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:32:13.337 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:32:13.331+0000 7f6f70db27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:32:13.338 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:32:13.335+0000 7f6f70db27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:32:13.338 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:32:13.335+0000 7f6f70db27c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-26T00:32:13.338 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:32:13.335+0000 7f6f70db27c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-26T00:32:13.339 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:32:13.335+0000 7f6f70db27c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-26T00:32:13.441 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:32:13.439+0000 7f6f70db27c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-26T00:32:21.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-26T00:32:21.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-26T00:32:21.458 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-26T00:32:21.459 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd0 key to auth repository 2024-03-26T00:32:21.459 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-26T00:32:21.845 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.0 2024-03-26T00:32:21.846 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-26T00:32:21.846 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-26T00:32:21.846 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-26T00:32:21.846 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-26T00:32:21.846 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-26T00:32:21.878 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:32:21.875+0000 7f2ebdefb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:32:21.932 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:32:21.927+0000 7f2ebdefb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:32:21.934 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:32:21.931+0000 7f2ebdefb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:32:22.152 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-26T00:32:22.152 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:32:22.152 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-26T00:32:22.152 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:32:22.152 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:32:22.152 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:32:22.152 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:32:22.152 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-26T00:32:22.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:32:22.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:32:22.445 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:32:22.516 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:32:22.511+0000 7f2ebdefb7c0 -1 Falling back to public interface 2024-03-26T00:32:23.447 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:32:23.447 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:32:23.448 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-26T00:32:23.448 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:32:23.448 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:32:23.448 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:32:23.769 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:32:24.770 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:32:24.771 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:32:24.771 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-26T00:32:24.771 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:32:24.771 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:32:24.772 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:32:25.089 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:32:25.385 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:32:25.383+0000 7f2ebdefb7c0 -1 osd.0 0 log_to_monitors true 2024-03-26T00:32:26.091 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:32:26.091 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:32:26.091 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-26T00:32:26.092 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:32:26.092 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:32:26.092 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:32:26.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:32:27.583 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:32:27.583 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:32:27.583 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-26T00:32:27.584 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-26T00:32:27.584 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:32:27.584 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:32:27.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:32:28.913 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:32:28.913 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:32:28.913 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-26T00:32:28.914 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-26T00:32:28.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:32:28.914 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:32:29.297 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:32:30.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:32:30.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:32:30.299 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-26T00:32:30.300 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-26T00:32:30.300 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:32:30.300 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:32:30.772 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:32:31.774 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:32:31.774 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:32:31.775 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-26T00:32:31.775 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-26T00:32:31.775 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:32:31.775 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:32:32.089 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:32:33.091 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:32:33.091 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:32:33.092 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-26T00:32:33.092 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-26T00:32:33.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:32:33.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:32:33.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:32:34.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:32:34.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:32:34.588 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-26T00:32:34.589 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-26T00:32:34.589 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:32:34.589 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:32:34.905 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:32:35.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:32:35.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:32:35.907 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-26T00:32:35.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-26T00:32:35.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:32:35.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:32:36.320 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:32:37.322 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:32:37.322 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:32:37.323 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-26T00:32:37.323 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-26T00:32:37.323 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:32:37.323 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:32:37.893 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:32:38.894 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:32:38.894 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:32:38.895 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-26T00:32:38.895 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-26T00:32:38.895 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:32:38.896 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:32:39.286 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:32:40.288 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:32:40.288 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:32:40.288 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-26T00:32:40.289 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-26T00:32:40.289 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:32:40.289 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:32:40.728 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:32:41.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:32:41.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:32:41.730 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-26T00:32:41.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-26T00:32:41.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:32:41.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:32:42.092 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:32:43.094 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:32:43.094 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:32:43.094 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-26T00:32:43.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-26T00:32:43.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:32:43.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:32:43.694 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:32:44.696 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:32:44.696 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:32:44.696 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-26T00:32:44.697 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-26T00:32:44.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:32:44.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:32:45.012 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:32:46.014 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:32:46.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:32:46.015 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-26T00:32:46.016 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-26T00:32:46.016 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:32:46.016 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:32:46.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:32:47.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:32:47.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:32:47.579 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-26T00:32:47.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-26T00:32:47.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:32:47.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:32:48.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:32:49.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:32:49.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:32:49.071 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-26T00:32:49.071 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-26T00:32:49.071 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:32:49.072 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:32:49.842 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:32:50.843 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:32:50.843 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:32:50.843 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-26T00:32:50.844 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-26T00:32:50.844 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:32:50.844 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:32:51.541 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:32:52.543 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:32:52.543 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:32:52.543 INFO:tasks.workunit.client.0.smithi129.stdout:21 2024-03-26T00:32:52.544 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-26T00:32:52.544 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:32:52.544 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:32:52.983 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:32:53.985 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:32:53.985 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:32:53.985 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-26T00:32:53.986 INFO:tasks.workunit.client.0.smithi129.stdout:22 2024-03-26T00:32:53.986 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:32:53.986 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:32:54.880 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:32:55.882 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:32:55.882 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:32:55.882 INFO:tasks.workunit.client.0.smithi129.stdout:23 2024-03-26T00:32:55.883 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-03-26T00:32:55.883 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:32:55.883 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:32:56.653 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:32:57.654 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:32:57.655 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:32:57.655 INFO:tasks.workunit.client.0.smithi129.stdout:24 2024-03-26T00:32:57.655 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-03-26T00:32:57.655 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:32:57.656 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:32:58.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:32:59.042 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:32:59.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:32:59.043 INFO:tasks.workunit.client.0.smithi129.stdout:25 2024-03-26T00:32:59.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-03-26T00:32:59.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:32:59.044 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:32:59.346 INFO:tasks.workunit.client.0.smithi129.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3497960021,v1:127.0.0.1:6803/3497960021] [v2:127.0.0.1:6804/3497960021,v1:127.0.0.1:6805/3497960021] exists,up ebf71738-9d30-40f8-aded-e8cf86f55f5e 2024-03-26T00:32:59.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:32:59.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:32:59.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:32:59.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: for id in $(seq 0 $(expr $OSDS - 1)) 2024-03-26T00:32:59.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:667: TEST_repair_stats: run_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-26T00:32:59.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-26T00:32:59.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-26T00:32:59.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-26T00:32:59.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-26T00:32:59.348 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:32:59.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:32:59.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:32:59.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:32:59.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:32:59.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:32:59.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:32:59.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-26T00:32:59.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-26T00:32:59.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:32:59.349 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-26T00:32:59.349 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:32:59.349 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:32:59.349 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:32:59.349 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:32:59.350 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:32:59.352 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:32:59.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:32:59.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:32:59.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-26T00:32:59.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-26T00:32:59.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:32:59.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:32:59.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:32:59.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:32:59.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:32:59.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:32:59.353 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-26T00:32:59.353 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-26T00:32:59.353 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:32:59.353 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-26T00:32:59.353 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bac5ce16-daaf-4fff-b019-3e6d1bd38425 2024-03-26T00:32:59.353 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 bac5ce16-daaf-4fff-b019-3e6d1bd38425' 2024-03-26T00:32:59.353 INFO:tasks.workunit.client.0.smithi129.stdout:add osd1 bac5ce16-daaf-4fff-b019-3e6d1bd38425 2024-03-26T00:32:59.353 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-26T00:32:59.369 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC7FwJmC2D/FRAAveRTuWTZiidkst4er9dhcw== 2024-03-26T00:32:59.369 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC7FwJmC2D/FRAAveRTuWTZiidkst4er9dhcw=="}' 2024-03-26T00:32:59.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bac5ce16-daaf-4fff-b019-3e6d1bd38425 -i td/osd-scrub-repair/1/new.json 2024-03-26T00:32:59.828 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-26T00:32:59.848 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-26T00:32:59.849 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQC7FwJmC2D/FRAAveRTuWTZiidkst4er9dhcw== --osd-uuid bac5ce16-daaf-4fff-b019-3e6d1bd38425 2024-03-26T00:32:59.881 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:32:59.878+0000 7fb6fb8947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:32:59.963 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:32:59.958+0000 7fb6fb8947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:32:59.965 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:32:59.958+0000 7fb6fb8947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:32:59.965 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:32:59.962+0000 7fb6fb8947c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-26T00:32:59.965 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:32:59.962+0000 7fb6fb8947c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-26T00:32:59.965 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:32:59.962+0000 7fb6fb8947c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-26T00:33:00.056 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:33:00.050+0000 7fb6fb8947c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-26T00:33:08.356 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-26T00:33:08.356 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-26T00:33:08.358 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd1 key to auth repository 2024-03-26T00:33:08.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-26T00:33:08.359 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-26T00:33:08.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-26T00:33:08.758 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.1 2024-03-26T00:33:08.759 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-26T00:33:08.759 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-26T00:33:08.759 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-26T00:33:08.759 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-26T00:33:08.791 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:33:08.786+0000 7f4ca34647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:33:08.826 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:33:08.822+0000 7f4ca34647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:33:08.828 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:33:08.822+0000 7f4ca34647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:33:09.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-26T00:33:09.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:33:09.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:33:09.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:33:09.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:33:09.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:33:09.113 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-26T00:33:09.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:33:09.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:33:09.113 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:33:09.415 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:33:09.410+0000 7f4ca34647c0 -1 Falling back to public interface 2024-03-26T00:33:09.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:33:10.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:33:10.495 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:33:10.495 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-26T00:33:10.495 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:33:10.495 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:33:10.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:33:10.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:33:11.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:33:11.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:33:11.805 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:33:11.806 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-26T00:33:11.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:33:11.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:33:12.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:33:12.556 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:33:12.550+0000 7f4ca34647c0 -1 osd.1 0 log_to_monitors true 2024-03-26T00:33:13.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:33:13.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:33:13.268 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-26T00:33:13.268 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:33:13.268 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:33:13.268 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:33:13.758 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:33:14.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:33:14.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:33:14.760 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-26T00:33:14.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-26T00:33:14.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:33:14.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:33:15.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:33:16.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:33:16.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:33:16.238 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-26T00:33:16.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-26T00:33:16.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:33:16.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:33:16.560 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:33:17.562 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:33:17.563 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:33:17.563 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-26T00:33:17.563 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-26T00:33:17.563 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:33:17.563 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:33:18.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:33:19.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:33:19.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:33:19.070 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-26T00:33:19.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-26T00:33:19.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:33:19.070 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:33:19.495 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:33:20.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:33:20.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:33:20.497 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-26T00:33:20.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-26T00:33:20.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:33:20.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:33:21.036 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:33:22.039 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:33:22.039 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:33:22.039 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-26T00:33:22.039 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-26T00:33:22.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:33:22.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:33:22.549 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:33:23.551 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:33:23.551 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:33:23.552 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-26T00:33:23.552 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-26T00:33:23.552 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:33:23.552 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:33:24.077 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:33:25.079 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:33:25.079 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:33:25.080 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-26T00:33:25.080 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-26T00:33:25.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:33:25.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:33:25.462 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:33:26.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:33:26.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:33:26.464 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-26T00:33:26.465 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-26T00:33:26.465 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:33:26.465 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:33:26.936 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:33:27.938 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:33:27.939 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:33:27.939 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-26T00:33:27.939 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-26T00:33:27.940 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:33:27.940 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:33:28.248 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:33:29.250 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:33:29.250 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:33:29.251 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-26T00:33:29.251 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-26T00:33:29.251 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:33:29.251 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:33:29.850 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:33:30.852 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:33:30.852 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:33:30.852 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-26T00:33:30.853 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-26T00:33:30.853 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:33:30.853 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:33:31.156 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:33:32.159 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:33:32.159 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:33:32.159 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-26T00:33:32.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-26T00:33:32.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:33:32.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:33:32.624 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:33:33.626 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:33:33.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:33:33.627 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-26T00:33:33.627 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-26T00:33:33.628 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:33:33.628 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:33:34.536 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:33:35.538 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:33:35.538 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:33:35.538 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-26T00:33:35.539 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-26T00:33:35.539 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:33:35.539 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:33:36.759 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:33:37.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:33:37.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:33:37.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-26T00:33:37.761 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-26T00:33:37.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:33:37.762 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:33:38.526 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:33:39.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:33:39.528 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:33:39.528 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-26T00:33:39.529 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-26T00:33:39.529 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:33:39.529 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:33:39.846 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:33:40.848 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:33:40.848 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:33:40.848 INFO:tasks.workunit.client.0.smithi129.stdout:21 2024-03-26T00:33:40.849 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-26T00:33:40.849 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:33:40.849 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:33:41.454 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:33:42.455 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:33:42.456 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:33:42.456 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-26T00:33:42.456 INFO:tasks.workunit.client.0.smithi129.stdout:22 2024-03-26T00:33:42.456 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:33:42.456 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:33:43.163 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:33:44.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:33:44.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:33:44.165 INFO:tasks.workunit.client.0.smithi129.stdout:23 2024-03-26T00:33:44.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-03-26T00:33:44.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:33:44.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:33:45.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:33:46.195 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:33:46.196 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:33:46.196 INFO:tasks.workunit.client.0.smithi129.stdout:24 2024-03-26T00:33:46.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-03-26T00:33:46.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:33:46.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:33:46.644 INFO:tasks.workunit.client.0.smithi129.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2557442212,v1:127.0.0.1:6811/2557442212] [v2:127.0.0.1:6812/2557442212,v1:127.0.0.1:6813/2557442212] exists,up bac5ce16-daaf-4fff-b019-3e6d1bd38425 2024-03-26T00:33:46.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:33:46.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:33:46.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:33:46.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:670: TEST_repair_stats: create_pool testpool 1 1 2024-03-26T00:33:46.645 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-03-26T00:33:47.192 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'testpool' created 2024-03-26T00:33:47.215 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-26T00:33:48.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:671: TEST_repair_stats: ceph osd pool set testpool size 2 2024-03-26T00:33:49.194 INFO:tasks.workunit.client.0.smithi129.stderr:set pool 1 size to 2 2024-03-26T00:33:49.220 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:672: TEST_repair_stats: wait_for_clean 2024-03-26T00:33:49.220 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:33:49.220 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:33:49.220 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:33:49.221 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:33:49.224 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:33:49.224 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:33:49.224 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:33:49.225 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:33:49.225 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:33:49.354 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:33:49.355 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:33:49.355 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:33:49.355 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:33:49.355 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:33:49.355 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:33:49.672 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:33:49.673 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:33:49.673 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:33:49.673 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:33:49.674 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:33:49.994 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836497 2024-03-26T00:33:49.994 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836497 2024-03-26T00:33:49.994 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497' 2024-03-26T00:33:49.994 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:33:49.995 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:33:50.159 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672965 2024-03-26T00:33:50.159 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672965 2024-03-26T00:33:50.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497 1-42949672965' 2024-03-26T00:33:50.160 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:33:50.160 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836497 2024-03-26T00:33:50.160 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:33:50.161 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:33:50.162 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836497 2024-03-26T00:33:50.162 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:33:50.163 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836497 2024-03-26T00:33:50.163 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2024-03-26T00:33:50.164 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 21474836497 2024-03-26T00:33:50.164 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:33:50.641 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836497 2024-03-26T00:33:50.641 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:33:51.642 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:33:51.643 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:33:51.965 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836497 2024-03-26T00:33:51.965 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:33:51.966 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:33:51.966 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672965 2024-03-26T00:33:51.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:33:51.968 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672965 2024-03-26T00:33:51.968 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:33:51.970 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672965 2024-03-26T00:33:51.970 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-03-26T00:33:51.973 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 42949672965 2024-03-26T00:33:51.974 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:33:52.264 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672965 -lt 42949672965 2024-03-26T00:33:52.265 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:33:52.266 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:33:52.266 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:33:52.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-26T00:33:52.665 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:33:52.665 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:33:52.666 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:33:52.666 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:33:52.666 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:33:52.666 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:33:52.666 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:33:53.033 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-26T00:33:53.033 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:33:53.034 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:33:53.034 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:33:53.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-03-26T00:33:53.393 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-03-26T00:33:53.393 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-26T00:33:53.393 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-03-26T00:33:53.393 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-26T00:33:53.393 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-26T00:33:53.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-26T00:33:53.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:33:53.495 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:33:53.495 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:33:53.496 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:33:53.496 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:33:53.496 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:33:53.496 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:33:53.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-26T00:33:53.807 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:33:53.809 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:33:53.810 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:33:54.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-26T00:33:54.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:33:54.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:33:54.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:675: TEST_repair_stats: local payload=ABCDEF 2024-03-26T00:33:54.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: echo ABCDEF 2024-03-26T00:33:54.168 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: seq 1 30 2024-03-26T00:33:54.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-26T00:33:54.169 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:33:54.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-26T00:33:54.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:33:54.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-26T00:33:54.546 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:33:54.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-26T00:33:54.755 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:33:54.939 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-26T00:33:54.939 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:33:55.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-26T00:33:55.124 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:33:55.315 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-26T00:33:55.315 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:33:55.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-26T00:33:55.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:33:55.683 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-26T00:33:55.683 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:33:55.883 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-26T00:33:55.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:33:56.101 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-26T00:33:56.101 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj11 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:33:56.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-26T00:33:56.319 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj12 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:33:56.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-26T00:33:56.511 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj13 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:33:56.695 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-26T00:33:56.695 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj14 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:33:56.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-26T00:33:56.888 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj15 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:33:57.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-26T00:33:57.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj16 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:33:57.288 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-26T00:33:57.288 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj17 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:33:57.602 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-26T00:33:57.602 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj18 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:33:57.924 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-26T00:33:57.924 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj19 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:33:58.132 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-26T00:33:58.133 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj20 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:33:58.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-26T00:33:58.326 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj21 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:33:58.525 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-26T00:33:58.525 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj22 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:33:58.709 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-26T00:33:58.709 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj23 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:33:58.902 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-26T00:33:58.902 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj24 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:33:59.094 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-26T00:33:59.094 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj25 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:33:59.286 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-26T00:33:59.286 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj26 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:33:59.503 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-26T00:33:59.503 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj27 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:33:59.695 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-26T00:33:59.695 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj28 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:33:59.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-26T00:33:59.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj29 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:34:00.105 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-26T00:34:00.106 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj30 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:34:00.299 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:684: TEST_repair_stats: get_not_primary testpool obj1 2024-03-26T00:34:00.299 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-26T00:34:00.300 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-03-26T00:34:00.300 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool obj1 2024-03-26T00:34:00.300 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-26T00:34:00.300 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-03-26T00:34:00.301 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-03-26T00:34:00.301 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-26T00:34:00.629 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-26T00:34:00.629 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool obj1 2024-03-26T00:34:00.629 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-26T00:34:00.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:684: TEST_repair_stats: local other=0 2024-03-26T00:34:00.949 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_repair_stats: get_pg testpool obj1 2024-03-26T00:34:00.949 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-03-26T00:34:00.949 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-03-26T00:34:00.950 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-03-26T00:34:00.950 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-26T00:34:01.334 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_repair_stats: local pgid=1.0 2024-03-26T00:34:01.335 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:686: TEST_repair_stats: get_primary testpool obj1 2024-03-26T00:34:01.335 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-26T00:34:01.335 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-03-26T00:34:01.336 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-03-26T00:34:01.336 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-26T00:34:01.629 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:686: TEST_repair_stats: local primary=1 2024-03-26T00:34:01.629 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:688: TEST_repair_stats: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-26T00:34:01.630 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:34:01.630 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:34:01.630 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:34:01.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:34:01.631 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:34:02.943 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:34:02.943 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:689: TEST_repair_stats: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-26T00:34:02.943 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:34:02.943 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:34:02.944 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:34:02.944 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:34:02.944 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:34:03.257 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:34:03.258 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: seq 1 20 2024-03-26T00:34:03.259 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-26T00:34:03.260 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 1 % 2 2024-03-26T00:34:03.262 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-26T00:34:03.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 remove 2024-03-26T00:34:03.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:34:03.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:34:03.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:34:03.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:34:03.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:34:03.263 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 remove 2024-03-26T00:34:05.000 INFO:tasks.workunit.client.0.smithi129.stdout:remove #1:ff7b1f36:::obj1:head# 2024-03-26T00:34:05.535 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-26T00:34:05.535 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 2 % 2 2024-03-26T00:34:05.537 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-26T00:34:05.537 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-03-26T00:34:05.537 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:34:05.537 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:34:05.538 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:34:05.538 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:34:05.538 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:34:05.538 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-03-26T00:34:07.254 INFO:tasks.workunit.client.0.smithi129.stdout:remove #1:104778fc:::obj2:head# 2024-03-26T00:34:07.788 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-26T00:34:07.789 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 3 % 2 2024-03-26T00:34:07.791 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-26T00:34:07.791 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj3 remove 2024-03-26T00:34:07.791 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:34:07.791 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:34:07.791 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:34:07.791 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:34:07.791 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:34:07.791 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj3 remove 2024-03-26T00:34:08.856 INFO:tasks.workunit.client.0.smithi129.stdout:remove #1:8dd16f86:::obj3:head# 2024-03-26T00:34:09.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-26T00:34:09.392 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 4 % 2 2024-03-26T00:34:09.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-26T00:34:09.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj4 remove 2024-03-26T00:34:09.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:34:09.394 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:34:09.395 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:34:09.395 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:34:09.395 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:34:09.395 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj4 remove 2024-03-26T00:34:10.601 INFO:tasks.workunit.client.0.smithi129.stdout:remove #1:0ee9ae15:::obj4:head# 2024-03-26T00:34:11.136 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-26T00:34:11.136 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 5 % 2 2024-03-26T00:34:11.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-26T00:34:11.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj5 remove 2024-03-26T00:34:11.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:34:11.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:34:11.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:34:11.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:34:11.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:34:11.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj5 remove 2024-03-26T00:34:12.270 INFO:tasks.workunit.client.0.smithi129.stdout:remove #1:c52c9666:::obj5:head# 2024-03-26T00:34:12.803 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-26T00:34:12.804 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 6 % 2 2024-03-26T00:34:12.806 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-26T00:34:12.807 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj6 remove 2024-03-26T00:34:12.807 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:34:12.807 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:34:12.807 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:34:12.807 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:34:12.807 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:34:12.807 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj6 remove 2024-03-26T00:34:13.822 INFO:tasks.workunit.client.0.smithi129.stdout:remove #1:233a42c1:::obj6:head# 2024-03-26T00:34:14.368 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-26T00:34:14.368 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 7 % 2 2024-03-26T00:34:14.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-26T00:34:14.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj7 remove 2024-03-26T00:34:14.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:34:14.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:34:14.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:34:14.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:34:14.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:34:14.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj7 remove 2024-03-26T00:34:15.299 INFO:tasks.workunit.client.0.smithi129.stdout:remove #1:9a09113a:::obj7:head# 2024-03-26T00:34:15.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-26T00:34:15.836 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 8 % 2 2024-03-26T00:34:15.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-26T00:34:15.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj8 remove 2024-03-26T00:34:15.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:34:15.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:34:15.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:34:15.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:34:15.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:34:15.838 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj8 remove 2024-03-26T00:34:16.768 INFO:tasks.workunit.client.0.smithi129.stdout:remove #1:a557efb1:::obj8:head# 2024-03-26T00:34:17.304 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-26T00:34:17.305 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 9 % 2 2024-03-26T00:34:17.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-26T00:34:17.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj9 remove 2024-03-26T00:34:17.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:34:17.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:34:17.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:34:17.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:34:17.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:34:17.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj9 remove 2024-03-26T00:34:18.287 INFO:tasks.workunit.client.0.smithi129.stdout:remove #1:fe351e27:::obj9:head# 2024-03-26T00:34:18.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-26T00:34:18.832 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 10 % 2 2024-03-26T00:34:18.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-26T00:34:18.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj10 remove 2024-03-26T00:34:18.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:34:18.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:34:18.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:34:18.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:34:18.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:34:18.835 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj10 remove 2024-03-26T00:34:19.781 INFO:tasks.workunit.client.0.smithi129.stdout:remove #1:d1337354:::obj10:head# 2024-03-26T00:34:20.320 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-26T00:34:20.320 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 11 % 2 2024-03-26T00:34:20.322 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-26T00:34:20.322 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj11 remove 2024-03-26T00:34:20.322 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:34:20.323 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:34:20.323 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:34:20.323 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:34:20.323 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:34:20.323 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj11 remove 2024-03-26T00:34:21.266 INFO:tasks.workunit.client.0.smithi129.stdout:remove #1:eeae2a94:::obj11:head# 2024-03-26T00:34:22.299 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-26T00:34:22.300 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 12 % 2 2024-03-26T00:34:22.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-26T00:34:22.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj12 remove 2024-03-26T00:34:22.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:34:22.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:34:22.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:34:22.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:34:22.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:34:22.303 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj12 remove 2024-03-26T00:34:23.195 INFO:tasks.workunit.client.0.smithi129.stdout:remove #1:af5f95cb:::obj12:head# 2024-03-26T00:34:24.252 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-26T00:34:24.253 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 13 % 2 2024-03-26T00:34:24.255 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-26T00:34:24.255 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj13 remove 2024-03-26T00:34:24.255 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:34:24.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:34:24.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:34:24.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:34:24.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:34:24.256 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj13 remove 2024-03-26T00:34:25.081 INFO:tasks.workunit.client.0.smithi129.stdout:remove #1:a61ad63e:::obj13:head# 2024-03-26T00:34:25.619 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-26T00:34:25.621 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 14 % 2 2024-03-26T00:34:25.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-26T00:34:25.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj14 remove 2024-03-26T00:34:25.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:34:25.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:34:25.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:34:25.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:34:25.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:34:25.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj14 remove 2024-03-26T00:34:26.550 INFO:tasks.workunit.client.0.smithi129.stdout:remove #1:e0a44829:::obj14:head# 2024-03-26T00:34:27.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-26T00:34:27.091 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 15 % 2 2024-03-26T00:34:27.094 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-26T00:34:27.094 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj15 remove 2024-03-26T00:34:27.094 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:34:27.094 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:34:27.094 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:34:27.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:34:27.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:34:27.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj15 remove 2024-03-26T00:34:27.995 INFO:tasks.workunit.client.0.smithi129.stdout:remove #1:ab946124:::obj15:head# 2024-03-26T00:34:28.530 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-26T00:34:28.530 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 16 % 2 2024-03-26T00:34:28.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-26T00:34:28.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj16 remove 2024-03-26T00:34:28.532 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:34:28.533 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:34:28.533 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:34:28.533 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:34:28.533 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:34:28.533 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj16 remove 2024-03-26T00:34:29.353 INFO:tasks.workunit.client.0.smithi129.stdout:remove #1:461f8b5e:::obj16:head# 2024-03-26T00:34:29.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-26T00:34:29.888 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 17 % 2 2024-03-26T00:34:29.890 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-26T00:34:29.890 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj17 remove 2024-03-26T00:34:29.890 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:34:29.890 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:34:29.890 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:34:29.890 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:34:29.891 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:34:29.891 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj17 remove 2024-03-26T00:34:30.731 INFO:tasks.workunit.client.0.smithi129.stdout:remove #1:24fc6e92:::obj17:head# 2024-03-26T00:34:31.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-26T00:34:31.268 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 18 % 2 2024-03-26T00:34:31.270 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-26T00:34:31.270 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj18 remove 2024-03-26T00:34:31.270 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:34:31.270 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:34:31.270 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:34:31.270 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:34:31.270 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:34:31.270 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj18 remove 2024-03-26T00:34:32.230 INFO:tasks.workunit.client.0.smithi129.stdout:remove #1:b9836a99:::obj18:head# 2024-03-26T00:34:32.766 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-26T00:34:32.767 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 19 % 2 2024-03-26T00:34:32.770 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-26T00:34:32.770 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj19 remove 2024-03-26T00:34:32.770 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:34:32.770 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:34:32.770 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:34:32.770 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:34:32.770 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:34:32.770 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj19 remove 2024-03-26T00:34:33.659 INFO:tasks.workunit.client.0.smithi129.stdout:remove #1:00b31b6c:::obj19:head# 2024-03-26T00:34:34.203 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-26T00:34:34.204 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 20 % 2 2024-03-26T00:34:34.206 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-26T00:34:34.206 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj20 remove 2024-03-26T00:34:34.206 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:34:34.206 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:34:34.206 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:34:34.206 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:34:34.206 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:34:34.206 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj20 remove 2024-03-26T00:34:35.036 INFO:tasks.workunit.client.0.smithi129.stdout:remove #1:24b5611b:::obj20:head# 2024-03-26T00:34:35.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:696: TEST_repair_stats: activate_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-26T00:34:35.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:34:35.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:34:35.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-26T00:34:35.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:34:35.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:34:35.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:34:35.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:34:35.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:34:35.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:34:35.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:34:35.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:34:35.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:34:35.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:34:35.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:34:35.582 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:34:35.582 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:34:35.583 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:34:35.583 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:34:35.583 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:34:35.583 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:34:35.584 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:34:35.584 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:34:35.584 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:34:35.584 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:34:35.584 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:34:35.584 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:34:35.584 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:34:35.584 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:34:35.584 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:34:35.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:34:35.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-26T00:34:35.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:34:35.585 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.1 2024-03-26T00:34:35.585 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-26T00:34:35.586 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-26T00:34:35.587 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-26T00:34:35.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-26T00:34:35.589 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:34:35.589 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:34:35.589 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:34:35.622 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:34:35.620+0000 7efc6b6337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:34:35.624 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:34:35.620+0000 7efc6b6337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:34:35.626 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:34:35.624+0000 7efc6b6337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:34:35.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-26T00:34:35.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:34:35.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:34:35.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:34:35.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:34:35.910 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:34:35.910 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-26T00:34:35.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:34:35.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:34:35.911 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:34:36.159 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:34:36.156+0000 7efc6b6337c0 -1 Falling back to public interface 2024-03-26T00:34:36.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:34:37.269 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:34:37.269 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:34:37.269 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-26T00:34:37.270 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:34:37.270 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:34:37.270 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:34:37.575 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:34:38.576 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:34:38.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:34:38.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:34:38.577 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-26T00:34:38.577 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:34:38.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:34:38.890 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:34:38.922 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:34:38.920+0000 7efc6b6337c0 -1 osd.1 18 log_to_monitors true 2024-03-26T00:34:39.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:34:39.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:34:39.892 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-26T00:34:39.893 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:34:39.893 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:34:39.893 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:34:40.263 INFO:tasks.workunit.client.0.smithi129.stdout:osd.1 up in weight 1 up_from 21 up_thru 21 down_at 19 last_clean_interval [10,18) [v2:127.0.0.1:6802/222189425,v1:127.0.0.1:6803/222189425] [v2:127.0.0.1:6804/222189425,v1:127.0.0.1:6805/222189425] exists,up bac5ce16-daaf-4fff-b019-3e6d1bd38425 2024-03-26T00:34:40.266 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:34:40.266 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:34:40.266 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:34:40.266 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:697: TEST_repair_stats: activate_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-26T00:34:40.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:34:40.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:34:40.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-26T00:34:40.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:34:40.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:34:40.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:34:40.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:34:40.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:34:40.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:34:40.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-26T00:34:40.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-26T00:34:40.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:34:40.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:34:40.267 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:34:40.268 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:34:40.268 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:34:40.268 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:34:40.268 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:34:40.268 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:34:40.268 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:34:40.268 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:34:40.268 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:34:40.268 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:34:40.268 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:34:40.268 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:34:40.269 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:34:40.269 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:34:40.269 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:34:40.269 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:34:40.269 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:34:40.269 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-26T00:34:40.269 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-26T00:34:40.269 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-26T00:34:40.270 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.0 2024-03-26T00:34:40.271 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-26T00:34:40.271 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-26T00:34:40.271 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-26T00:34:40.271 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:34:40.271 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:34:40.271 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:34:40.299 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:34:40.296+0000 7f7fc52d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:34:40.403 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:34:40.400+0000 7f7fc52d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:34:40.404 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:34:40.400+0000 7f7fc52d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:34:40.622 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-26T00:34:40.622 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:34:40.622 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-26T00:34:40.622 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:34:40.622 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:34:40.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:34:40.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:34:40.623 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-26T00:34:40.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:34:40.623 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:34:40.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:34:40.960 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:34:40.956+0000 7f7fc52d67c0 -1 Falling back to public interface 2024-03-26T00:34:41.932 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:34:41.932 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:34:41.933 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-26T00:34:41.933 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:34:41.933 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:34:41.933 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:34:42.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:34:43.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:34:43.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:34:43.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:34:43.238 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-26T00:34:43.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:34:43.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:34:43.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:34:43.807 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:34:43.804+0000 7f7fc52d67c0 -1 osd.0 16 log_to_monitors true 2024-03-26T00:34:44.566 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:34:44.566 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:34:44.567 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-26T00:34:44.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:34:44.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:34:44.567 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:34:44.891 INFO:tasks.workunit.client.0.smithi129.stdout:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 17 last_clean_interval [5,16) [v2:127.0.0.1:6810/264812859,v1:127.0.0.1:6811/264812859] [v2:127.0.0.1:6812/264812859,v1:127.0.0.1:6813/264812859] exists,up ebf71738-9d30-40f8-aded-e8cf86f55f5e 2024-03-26T00:34:44.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:34:44.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:34:44.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:34:44.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:698: TEST_repair_stats: wait_for_clean 2024-03-26T00:34:44.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:34:44.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:34:44.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:34:44.892 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:34:44.894 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:34:44.894 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:34:44.894 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:34:44.894 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:34:44.894 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:34:45.015 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:34:45.016 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:34:45.016 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:34:45.016 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:34:45.016 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:34:45.016 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:34:45.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:34:45.331 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:34:45.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:34:45.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:34:45.332 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:34:45.482 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215106 2024-03-26T00:34:45.482 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215106 2024-03-26T00:34:45.482 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215106' 2024-03-26T00:34:45.482 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:34:45.482 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:34:45.737 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=90194313219 2024-03-26T00:34:45.738 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 90194313219 2024-03-26T00:34:45.738 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215106 1-90194313219' 2024-03-26T00:34:45.738 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:34:45.739 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-103079215106 2024-03-26T00:34:45.739 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:34:45.739 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:34:45.740 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-103079215106 2024-03-26T00:34:45.740 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:34:45.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215106 2024-03-26T00:34:45.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2024-03-26T00:34:45.742 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 103079215106 2024-03-26T00:34:45.742 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:34:46.053 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 103079215106 2024-03-26T00:34:46.054 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:34:47.055 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:34:47.056 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:34:47.361 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 103079215106 2024-03-26T00:34:47.361 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:34:48.362 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-26T00:34:48.364 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:34:48.688 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215106 -lt 103079215106 2024-03-26T00:34:48.689 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:34:48.690 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:34:48.690 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-90194313219 2024-03-26T00:34:48.691 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:34:48.692 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-90194313219 2024-03-26T00:34:48.692 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:34:48.694 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=90194313219 2024-03-26T00:34:48.694 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 90194313219' 2024-03-26T00:34:48.694 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 90194313219 2024-03-26T00:34:48.694 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:34:49.028 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 90194313219 -lt 90194313219 2024-03-26T00:34:49.028 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:34:49.029 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:34:49.029 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:34:49.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-26T00:34:49.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:34:49.393 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:34:49.393 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:34:49.393 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:34:49.393 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:34:49.393 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:34:49.394 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:34:49.787 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-26T00:34:49.787 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:34:49.788 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:34:49.788 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:34:50.147 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-26T00:34:50.147 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:34:50.147 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:34:50.147 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:700: TEST_repair_stats: repair 1.0 2024-03-26T00:34:50.147 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-03-26T00:34:50.148 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-03-26T00:34:50.148 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-26T00:34:50.148 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:34:50.148 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:34:50.148 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-26T00:34:50.436 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-26T00:33:47.024690+0000 2024-03-26T00:34:50.436 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-03-26T00:34:50.713 INFO:tasks.workunit.client.0.smithi129.stderr:instructing pg 1.0 on osd.1 to repair 2024-03-26T00:34:50.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-03-26T00:33:47.024690+0000 2024-03-26T00:34:50.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-03-26T00:34:50.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-26T00:33:47.024690+0000 2024-03-26T00:34:50.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-26T00:34:50.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-26T00:34:50.730 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:34:50.730 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-26T00:34:50.730 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-26T00:34:50.731 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:34:50.731 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:34:50.731 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-26T00:34:51.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:33:47.024690+0000 '>' 2024-03-26T00:33:47.024690+0000 2024-03-26T00:34:51.011 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:34:52.012 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:34:52.012 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:34:52.014 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-26T00:34:52.014 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-26T00:34:52.014 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:34:52.014 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:34:52.014 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-26T00:34:52.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:33:47.024690+0000 '>' 2024-03-26T00:33:47.024690+0000 2024-03-26T00:34:52.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:34:53.414 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:34:53.415 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:34:53.415 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-26T00:34:53.415 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-26T00:34:53.415 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:34:53.416 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:34:53.416 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-26T00:34:53.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:33:47.024690+0000 '>' 2024-03-26T00:33:47.024690+0000 2024-03-26T00:34:53.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:34:54.736 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:34:54.736 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:34:54.737 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-26T00:34:54.737 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-26T00:34:54.737 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:34:54.738 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:34:54.738 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-26T00:34:55.057 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:33:47.024690+0000 '>' 2024-03-26T00:33:47.024690+0000 2024-03-26T00:34:55.057 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:34:56.058 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:34:56.058 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:34:56.060 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-26T00:34:56.060 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-26T00:34:56.060 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:34:56.060 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:34:56.061 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-26T00:34:56.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:34:51.187825+0000 '>' 2024-03-26T00:33:47.024690+0000 2024-03-26T00:34:56.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-26T00:34:56.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:701: TEST_repair_stats: wait_for_clean 2024-03-26T00:34:56.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:34:56.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:34:56.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:34:56.370 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:34:56.371 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:34:56.371 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:34:56.371 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:34:56.371 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:34:56.371 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:34:56.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:34:56.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:34:56.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:34:56.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:34:56.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:34:56.485 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:34:56.796 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:34:56.796 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:34:56.796 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:34:56.796 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:34:56.797 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:34:56.975 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215109 2024-03-26T00:34:56.975 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215109 2024-03-26T00:34:56.975 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215109' 2024-03-26T00:34:56.975 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:34:56.976 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:34:57.161 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=90194313222 2024-03-26T00:34:57.161 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 90194313222 2024-03-26T00:34:57.161 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215109 1-90194313222' 2024-03-26T00:34:57.161 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:34:57.162 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-103079215109 2024-03-26T00:34:57.162 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:34:57.163 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:34:57.164 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-103079215109 2024-03-26T00:34:57.164 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:34:57.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215109 2024-03-26T00:34:57.165 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 103079215109 2024-03-26T00:34:57.166 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 103079215109' 2024-03-26T00:34:57.166 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:34:57.471 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215108 -lt 103079215109 2024-03-26T00:34:57.471 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:34:58.473 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:34:58.474 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:34:58.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215109 -lt 103079215109 2024-03-26T00:34:58.782 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:34:58.784 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-90194313222 2024-03-26T00:34:58.784 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:34:58.785 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:34:58.786 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-90194313222 2024-03-26T00:34:58.786 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:34:58.788 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=90194313222 2024-03-26T00:34:58.788 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 90194313222' 2024-03-26T00:34:58.788 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 90194313222 2024-03-26T00:34:58.788 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:34:59.075 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 90194313222 -lt 90194313222 2024-03-26T00:34:59.076 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:34:59.076 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:34:59.076 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:34:59.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-26T00:34:59.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:34:59.435 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:34:59.435 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:34:59.435 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:34:59.435 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:34:59.436 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:34:59.436 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:34:59.785 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-26T00:34:59.786 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:34:59.786 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:34:59.786 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:35:00.149 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-26T00:35:00.149 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:35:00.149 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:35:00.149 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:702: TEST_repair_stats: ceph pg dump pgs 2024-03-26T00:35:00.459 INFO:tasks.workunit.client.0.smithi129.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-26T00:35:00.459 INFO:tasks.workunit.client.0.smithi129.stdout:1.0 30 0 0 0 0 210 0 0 30 0 30 active+clean 2024-03-26T00:34:52.623422+0000 16'30 25:124 [1,0] 1 [1,0] 1 16'30 2024-03-26T00:34:51.187825+0000 16'30 2024-03-26T00:34:51.187825+0000 0 1 periodic scrub scheduled @ 2024-03-27T00:34:51.187825+0000 20 0 2024-03-26T00:35:00.459 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-26T00:35:00.459 INFO:tasks.workunit.client.0.smithi129.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-26T00:35:00.459 INFO:tasks.workunit.client.0.smithi129.stderr:dumped pgs 2024-03-26T00:35:00.475 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:703: TEST_repair_stats: flush_pg_stats 2024-03-26T00:35:00.475 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:35:00.476 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:35:00.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:35:00.768 INFO:tasks.workunit.client.0.smithi129.stderr:1' 2024-03-26T00:35:00.768 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:35:00.768 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:35:00.768 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:35:00.900 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215111 2024-03-26T00:35:00.900 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215111 2024-03-26T00:35:00.900 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215111' 2024-03-26T00:35:00.900 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:35:00.901 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:35:01.072 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=90194313224 2024-03-26T00:35:01.073 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 90194313224 2024-03-26T00:35:01.073 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215111 1-90194313224' 2024-03-26T00:35:01.073 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:35:01.073 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:35:01.073 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-103079215111 2024-03-26T00:35:01.074 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:35:01.076 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-103079215111 2024-03-26T00:35:01.076 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:35:01.077 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215111 2024-03-26T00:35:01.077 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 103079215111 2024-03-26T00:35:01.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 103079215111' 2024-03-26T00:35:01.078 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:35:01.368 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215110 -lt 103079215111 2024-03-26T00:35:01.368 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:35:02.370 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:35:02.371 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:35:02.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215111 -lt 103079215111 2024-03-26T00:35:02.676 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:35:02.677 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-90194313224 2024-03-26T00:35:02.677 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:35:02.679 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:35:02.680 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-90194313224 2024-03-26T00:35:02.680 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:35:02.681 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=90194313224 2024-03-26T00:35:02.682 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 90194313224' 2024-03-26T00:35:02.682 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 90194313224 2024-03-26T00:35:02.682 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:35:03.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 90194313224 -lt 90194313224 2024-03-26T00:35:03.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:706: TEST_repair_stats: ceph pg 1.0 query 2024-03-26T00:35:03.028 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:706: TEST_repair_stats: jq .info.stats.stat_sum 2024-03-26T00:35:03.161 INFO:tasks.workunit.client.0.smithi129.stdout:{ 2024-03-26T00:35:03.161 INFO:tasks.workunit.client.0.smithi129.stdout: "num_bytes": 210, 2024-03-26T00:35:03.161 INFO:tasks.workunit.client.0.smithi129.stdout: "num_objects": 30, 2024-03-26T00:35:03.161 INFO:tasks.workunit.client.0.smithi129.stdout: "num_object_clones": 0, 2024-03-26T00:35:03.161 INFO:tasks.workunit.client.0.smithi129.stdout: "num_object_copies": 60, 2024-03-26T00:35:03.161 INFO:tasks.workunit.client.0.smithi129.stdout: "num_objects_missing_on_primary": 0, 2024-03-26T00:35:03.162 INFO:tasks.workunit.client.0.smithi129.stdout: "num_objects_missing": 0, 2024-03-26T00:35:03.162 INFO:tasks.workunit.client.0.smithi129.stdout: "num_objects_degraded": 0, 2024-03-26T00:35:03.162 INFO:tasks.workunit.client.0.smithi129.stdout: "num_objects_misplaced": 0, 2024-03-26T00:35:03.162 INFO:tasks.workunit.client.0.smithi129.stdout: "num_objects_unfound": 0, 2024-03-26T00:35:03.162 INFO:tasks.workunit.client.0.smithi129.stdout: "num_objects_dirty": 30, 2024-03-26T00:35:03.162 INFO:tasks.workunit.client.0.smithi129.stdout: "num_whiteouts": 0, 2024-03-26T00:35:03.162 INFO:tasks.workunit.client.0.smithi129.stdout: "num_read": 0, 2024-03-26T00:35:03.162 INFO:tasks.workunit.client.0.smithi129.stdout: "num_read_kb": 0, 2024-03-26T00:35:03.162 INFO:tasks.workunit.client.0.smithi129.stdout: "num_write": 30, 2024-03-26T00:35:03.162 INFO:tasks.workunit.client.0.smithi129.stdout: "num_write_kb": 30, 2024-03-26T00:35:03.162 INFO:tasks.workunit.client.0.smithi129.stdout: "num_scrub_errors": 0, 2024-03-26T00:35:03.162 INFO:tasks.workunit.client.0.smithi129.stdout: "num_shallow_scrub_errors": 0, 2024-03-26T00:35:03.163 INFO:tasks.workunit.client.0.smithi129.stdout: "num_deep_scrub_errors": 0, 2024-03-26T00:35:03.163 INFO:tasks.workunit.client.0.smithi129.stdout: "num_objects_recovered": 20, 2024-03-26T00:35:03.163 INFO:tasks.workunit.client.0.smithi129.stdout: "num_bytes_recovered": 140, 2024-03-26T00:35:03.163 INFO:tasks.workunit.client.0.smithi129.stdout: "num_keys_recovered": 0, 2024-03-26T00:35:03.163 INFO:tasks.workunit.client.0.smithi129.stdout: "num_objects_omap": 0, 2024-03-26T00:35:03.163 INFO:tasks.workunit.client.0.smithi129.stdout: "num_objects_hit_set_archive": 0, 2024-03-26T00:35:03.163 INFO:tasks.workunit.client.0.smithi129.stdout: "num_bytes_hit_set_archive": 0, 2024-03-26T00:35:03.163 INFO:tasks.workunit.client.0.smithi129.stdout: "num_flush": 0, 2024-03-26T00:35:03.163 INFO:tasks.workunit.client.0.smithi129.stdout: "num_flush_kb": 0, 2024-03-26T00:35:03.163 INFO:tasks.workunit.client.0.smithi129.stdout: "num_evict": 0, 2024-03-26T00:35:03.163 INFO:tasks.workunit.client.0.smithi129.stdout: "num_evict_kb": 0, 2024-03-26T00:35:03.163 INFO:tasks.workunit.client.0.smithi129.stdout: "num_promote": 0, 2024-03-26T00:35:03.164 INFO:tasks.workunit.client.0.smithi129.stdout: "num_flush_mode_high": 0, 2024-03-26T00:35:03.164 INFO:tasks.workunit.client.0.smithi129.stdout: "num_flush_mode_low": 0, 2024-03-26T00:35:03.164 INFO:tasks.workunit.client.0.smithi129.stdout: "num_evict_mode_some": 0, 2024-03-26T00:35:03.164 INFO:tasks.workunit.client.0.smithi129.stdout: "num_evict_mode_full": 0, 2024-03-26T00:35:03.164 INFO:tasks.workunit.client.0.smithi129.stdout: "num_objects_pinned": 0, 2024-03-26T00:35:03.164 INFO:tasks.workunit.client.0.smithi129.stdout: "num_legacy_snapsets": 0, 2024-03-26T00:35:03.164 INFO:tasks.workunit.client.0.smithi129.stdout: "num_large_omap_objects": 0, 2024-03-26T00:35:03.164 INFO:tasks.workunit.client.0.smithi129.stdout: "num_objects_manifest": 0, 2024-03-26T00:35:03.164 INFO:tasks.workunit.client.0.smithi129.stdout: "num_omap_bytes": 0, 2024-03-26T00:35:03.164 INFO:tasks.workunit.client.0.smithi129.stdout: "num_omap_keys": 0, 2024-03-26T00:35:03.164 INFO:tasks.workunit.client.0.smithi129.stdout: "num_objects_repaired": 20 2024-03-26T00:35:03.164 INFO:tasks.workunit.client.0.smithi129.stdout:} 2024-03-26T00:35:03.165 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: ceph pg 1.0 query 2024-03-26T00:35:03.165 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: jq .info.stats.stat_sum.num_objects_repaired 2024-03-26T00:35:03.296 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: COUNT=20 2024-03-26T00:35:03.296 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:708: TEST_repair_stats: test 20 = 20 2024-03-26T00:35:03.296 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:710: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-03-26T00:35:03.296 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:710: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 1 )' 2024-03-26T00:35:03.573 INFO:tasks.workunit.client.0.smithi129.stderr:dumped all 2024-03-26T00:35:03.588 INFO:tasks.workunit.client.0.smithi129.stdout:{ 2024-03-26T00:35:03.588 INFO:tasks.workunit.client.0.smithi129.stdout: "osd": 1, 2024-03-26T00:35:03.588 INFO:tasks.workunit.client.0.smithi129.stdout: "up_from": 21, 2024-03-26T00:35:03.589 INFO:tasks.workunit.client.0.smithi129.stdout: "seq": 90194313224, 2024-03-26T00:35:03.589 INFO:tasks.workunit.client.0.smithi129.stdout: "num_pgs": 1, 2024-03-26T00:35:03.589 INFO:tasks.workunit.client.0.smithi129.stdout: "num_osds": 1, 2024-03-26T00:35:03.589 INFO:tasks.workunit.client.0.smithi129.stdout: "num_per_pool_osds": 1, 2024-03-26T00:35:03.589 INFO:tasks.workunit.client.0.smithi129.stdout: "num_per_pool_omap_osds": 1, 2024-03-26T00:35:03.589 INFO:tasks.workunit.client.0.smithi129.stdout: "kb": 104857600, 2024-03-26T00:35:03.589 INFO:tasks.workunit.client.0.smithi129.stdout: "kb_used": 28560, 2024-03-26T00:35:03.589 INFO:tasks.workunit.client.0.smithi129.stdout: "kb_used_data": 328, 2024-03-26T00:35:03.589 INFO:tasks.workunit.client.0.smithi129.stdout: "kb_used_omap": 4, 2024-03-26T00:35:03.589 INFO:tasks.workunit.client.0.smithi129.stdout: "kb_used_meta": 28219, 2024-03-26T00:35:03.589 INFO:tasks.workunit.client.0.smithi129.stdout: "kb_avail": 104829040, 2024-03-26T00:35:03.589 INFO:tasks.workunit.client.0.smithi129.stdout: "statfs": { 2024-03-26T00:35:03.590 INFO:tasks.workunit.client.0.smithi129.stdout: "total": 107374182400, 2024-03-26T00:35:03.590 INFO:tasks.workunit.client.0.smithi129.stdout: "available": 107344936960, 2024-03-26T00:35:03.590 INFO:tasks.workunit.client.0.smithi129.stdout: "internally_reserved": 0, 2024-03-26T00:35:03.590 INFO:tasks.workunit.client.0.smithi129.stdout: "allocated": 335872, 2024-03-26T00:35:03.590 INFO:tasks.workunit.client.0.smithi129.stdout: "data_stored": 56243, 2024-03-26T00:35:03.590 INFO:tasks.workunit.client.0.smithi129.stdout: "data_compressed": 0, 2024-03-26T00:35:03.590 INFO:tasks.workunit.client.0.smithi129.stdout: "data_compressed_allocated": 0, 2024-03-26T00:35:03.590 INFO:tasks.workunit.client.0.smithi129.stdout: "data_compressed_original": 0, 2024-03-26T00:35:03.590 INFO:tasks.workunit.client.0.smithi129.stdout: "omap_allocated": 4398, 2024-03-26T00:35:03.590 INFO:tasks.workunit.client.0.smithi129.stdout: "internal_metadata": 28896978 2024-03-26T00:35:03.590 INFO:tasks.workunit.client.0.smithi129.stdout: }, 2024-03-26T00:35:03.590 INFO:tasks.workunit.client.0.smithi129.stdout: "hb_peers": [ 2024-03-26T00:35:03.590 INFO:tasks.workunit.client.0.smithi129.stdout: 0 2024-03-26T00:35:03.590 INFO:tasks.workunit.client.0.smithi129.stdout: ], 2024-03-26T00:35:03.591 INFO:tasks.workunit.client.0.smithi129.stdout: "snap_trim_queue_len": 0, 2024-03-26T00:35:03.591 INFO:tasks.workunit.client.0.smithi129.stdout: "num_snap_trimming": 0, 2024-03-26T00:35:03.591 INFO:tasks.workunit.client.0.smithi129.stdout: "num_shards_repaired": 10, 2024-03-26T00:35:03.591 INFO:tasks.workunit.client.0.smithi129.stdout: "op_queue_age_hist": { 2024-03-26T00:35:03.591 INFO:tasks.workunit.client.0.smithi129.stdout: "histogram": [], 2024-03-26T00:35:03.591 INFO:tasks.workunit.client.0.smithi129.stdout: "upper_bound": 1 2024-03-26T00:35:03.591 INFO:tasks.workunit.client.0.smithi129.stdout: }, 2024-03-26T00:35:03.591 INFO:tasks.workunit.client.0.smithi129.stdout: "perf_stat": { 2024-03-26T00:35:03.591 INFO:tasks.workunit.client.0.smithi129.stdout: "commit_latency_ms": 0, 2024-03-26T00:35:03.591 INFO:tasks.workunit.client.0.smithi129.stdout: "apply_latency_ms": 0, 2024-03-26T00:35:03.591 INFO:tasks.workunit.client.0.smithi129.stdout: "commit_latency_ns": 0, 2024-03-26T00:35:03.591 INFO:tasks.workunit.client.0.smithi129.stdout: "apply_latency_ns": 0 2024-03-26T00:35:03.591 INFO:tasks.workunit.client.0.smithi129.stdout: }, 2024-03-26T00:35:03.591 INFO:tasks.workunit.client.0.smithi129.stdout: "alerts": [] 2024-03-26T00:35:03.591 INFO:tasks.workunit.client.0.smithi129.stdout:} 2024-03-26T00:35:03.592 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-03-26T00:35:03.592 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 1 ).num_shards_repaired' 2024-03-26T00:35:03.858 INFO:tasks.workunit.client.0.smithi129.stderr:dumped all 2024-03-26T00:35:03.874 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: COUNT=10 2024-03-26T00:35:03.876 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:712: TEST_repair_stats: expr 20 / 2 2024-03-26T00:35:03.878 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:712: TEST_repair_stats: test 10 = 10 2024-03-26T00:35:03.878 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:714: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-03-26T00:35:03.878 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:714: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 0 )' 2024-03-26T00:35:04.191 INFO:tasks.workunit.client.0.smithi129.stderr:dumped all 2024-03-26T00:35:04.208 INFO:tasks.workunit.client.0.smithi129.stdout:{ 2024-03-26T00:35:04.208 INFO:tasks.workunit.client.0.smithi129.stdout: "osd": 0, 2024-03-26T00:35:04.208 INFO:tasks.workunit.client.0.smithi129.stdout: "up_from": 24, 2024-03-26T00:35:04.208 INFO:tasks.workunit.client.0.smithi129.stdout: "seq": 103079215111, 2024-03-26T00:35:04.208 INFO:tasks.workunit.client.0.smithi129.stdout: "num_pgs": 1, 2024-03-26T00:35:04.209 INFO:tasks.workunit.client.0.smithi129.stdout: "num_osds": 1, 2024-03-26T00:35:04.209 INFO:tasks.workunit.client.0.smithi129.stdout: "num_per_pool_osds": 1, 2024-03-26T00:35:04.209 INFO:tasks.workunit.client.0.smithi129.stdout: "num_per_pool_omap_osds": 1, 2024-03-26T00:35:04.209 INFO:tasks.workunit.client.0.smithi129.stdout: "kb": 104857600, 2024-03-26T00:35:04.209 INFO:tasks.workunit.client.0.smithi129.stdout: "kb_used": 28560, 2024-03-26T00:35:04.209 INFO:tasks.workunit.client.0.smithi129.stdout: "kb_used_data": 328, 2024-03-26T00:35:04.209 INFO:tasks.workunit.client.0.smithi129.stdout: "kb_used_omap": 4, 2024-03-26T00:35:04.209 INFO:tasks.workunit.client.0.smithi129.stdout: "kb_used_meta": 28219, 2024-03-26T00:35:04.209 INFO:tasks.workunit.client.0.smithi129.stdout: "kb_avail": 104829040, 2024-03-26T00:35:04.209 INFO:tasks.workunit.client.0.smithi129.stdout: "statfs": { 2024-03-26T00:35:04.209 INFO:tasks.workunit.client.0.smithi129.stdout: "total": 107374182400, 2024-03-26T00:35:04.209 INFO:tasks.workunit.client.0.smithi129.stdout: "available": 107344936960, 2024-03-26T00:35:04.209 INFO:tasks.workunit.client.0.smithi129.stdout: "internally_reserved": 0, 2024-03-26T00:35:04.209 INFO:tasks.workunit.client.0.smithi129.stdout: "allocated": 335872, 2024-03-26T00:35:04.210 INFO:tasks.workunit.client.0.smithi129.stdout: "data_stored": 56243, 2024-03-26T00:35:04.210 INFO:tasks.workunit.client.0.smithi129.stdout: "data_compressed": 0, 2024-03-26T00:35:04.210 INFO:tasks.workunit.client.0.smithi129.stdout: "data_compressed_allocated": 0, 2024-03-26T00:35:04.210 INFO:tasks.workunit.client.0.smithi129.stdout: "data_compressed_original": 0, 2024-03-26T00:35:04.210 INFO:tasks.workunit.client.0.smithi129.stdout: "omap_allocated": 4401, 2024-03-26T00:35:04.210 INFO:tasks.workunit.client.0.smithi129.stdout: "internal_metadata": 28896975 2024-03-26T00:35:04.210 INFO:tasks.workunit.client.0.smithi129.stdout: }, 2024-03-26T00:35:04.210 INFO:tasks.workunit.client.0.smithi129.stdout: "hb_peers": [ 2024-03-26T00:35:04.210 INFO:tasks.workunit.client.0.smithi129.stdout: 1 2024-03-26T00:35:04.210 INFO:tasks.workunit.client.0.smithi129.stdout: ], 2024-03-26T00:35:04.210 INFO:tasks.workunit.client.0.smithi129.stdout: "snap_trim_queue_len": 0, 2024-03-26T00:35:04.210 INFO:tasks.workunit.client.0.smithi129.stdout: "num_snap_trimming": 0, 2024-03-26T00:35:04.210 INFO:tasks.workunit.client.0.smithi129.stdout: "num_shards_repaired": 10, 2024-03-26T00:35:04.211 INFO:tasks.workunit.client.0.smithi129.stdout: "op_queue_age_hist": { 2024-03-26T00:35:04.211 INFO:tasks.workunit.client.0.smithi129.stdout: "histogram": [], 2024-03-26T00:35:04.211 INFO:tasks.workunit.client.0.smithi129.stdout: "upper_bound": 1 2024-03-26T00:35:04.211 INFO:tasks.workunit.client.0.smithi129.stdout: }, 2024-03-26T00:35:04.211 INFO:tasks.workunit.client.0.smithi129.stdout: "perf_stat": { 2024-03-26T00:35:04.211 INFO:tasks.workunit.client.0.smithi129.stdout: "commit_latency_ms": 0, 2024-03-26T00:35:04.211 INFO:tasks.workunit.client.0.smithi129.stdout: "apply_latency_ms": 0, 2024-03-26T00:35:04.211 INFO:tasks.workunit.client.0.smithi129.stdout: "commit_latency_ns": 0, 2024-03-26T00:35:04.211 INFO:tasks.workunit.client.0.smithi129.stdout: "apply_latency_ns": 0 2024-03-26T00:35:04.211 INFO:tasks.workunit.client.0.smithi129.stdout: }, 2024-03-26T00:35:04.211 INFO:tasks.workunit.client.0.smithi129.stdout: "alerts": [] 2024-03-26T00:35:04.211 INFO:tasks.workunit.client.0.smithi129.stdout:} 2024-03-26T00:35:04.212 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-03-26T00:35:04.212 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 0 ).num_shards_repaired' 2024-03-26T00:35:04.486 INFO:tasks.workunit.client.0.smithi129.stderr:dumped all 2024-03-26T00:35:04.501 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: COUNT=10 2024-03-26T00:35:04.502 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:716: TEST_repair_stats: expr 20 / 2 2024-03-26T00:35:04.503 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:716: TEST_repair_stats: test 10 = 10 2024-03-26T00:35:04.503 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:718: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-03-26T00:35:04.503 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:718: TEST_repair_stats: jq .pg_map.osd_stats_sum 2024-03-26T00:35:04.773 INFO:tasks.workunit.client.0.smithi129.stderr:dumped all 2024-03-26T00:35:04.789 INFO:tasks.workunit.client.0.smithi129.stdout:{ 2024-03-26T00:35:04.789 INFO:tasks.workunit.client.0.smithi129.stdout: "up_from": 0, 2024-03-26T00:35:04.789 INFO:tasks.workunit.client.0.smithi129.stdout: "seq": 0, 2024-03-26T00:35:04.789 INFO:tasks.workunit.client.0.smithi129.stdout: "num_pgs": 2, 2024-03-26T00:35:04.789 INFO:tasks.workunit.client.0.smithi129.stdout: "num_osds": 2, 2024-03-26T00:35:04.789 INFO:tasks.workunit.client.0.smithi129.stdout: "num_per_pool_osds": 2, 2024-03-26T00:35:04.789 INFO:tasks.workunit.client.0.smithi129.stdout: "num_per_pool_omap_osds": 2, 2024-03-26T00:35:04.789 INFO:tasks.workunit.client.0.smithi129.stdout: "kb": 209715200, 2024-03-26T00:35:04.790 INFO:tasks.workunit.client.0.smithi129.stdout: "kb_used": 57120, 2024-03-26T00:35:04.790 INFO:tasks.workunit.client.0.smithi129.stdout: "kb_used_data": 656, 2024-03-26T00:35:04.790 INFO:tasks.workunit.client.0.smithi129.stdout: "kb_used_omap": 8, 2024-03-26T00:35:04.790 INFO:tasks.workunit.client.0.smithi129.stdout: "kb_used_meta": 56439, 2024-03-26T00:35:04.790 INFO:tasks.workunit.client.0.smithi129.stdout: "kb_avail": 209658080, 2024-03-26T00:35:04.790 INFO:tasks.workunit.client.0.smithi129.stdout: "statfs": { 2024-03-26T00:35:04.790 INFO:tasks.workunit.client.0.smithi129.stdout: "total": 214748364800, 2024-03-26T00:35:04.790 INFO:tasks.workunit.client.0.smithi129.stdout: "available": 214689873920, 2024-03-26T00:35:04.790 INFO:tasks.workunit.client.0.smithi129.stdout: "internally_reserved": 0, 2024-03-26T00:35:04.790 INFO:tasks.workunit.client.0.smithi129.stdout: "allocated": 671744, 2024-03-26T00:35:04.790 INFO:tasks.workunit.client.0.smithi129.stdout: "data_stored": 112486, 2024-03-26T00:35:04.791 INFO:tasks.workunit.client.0.smithi129.stdout: "data_compressed": 0, 2024-03-26T00:35:04.791 INFO:tasks.workunit.client.0.smithi129.stdout: "data_compressed_allocated": 0, 2024-03-26T00:35:04.791 INFO:tasks.workunit.client.0.smithi129.stdout: "data_compressed_original": 0, 2024-03-26T00:35:04.791 INFO:tasks.workunit.client.0.smithi129.stdout: "omap_allocated": 8799, 2024-03-26T00:35:04.791 INFO:tasks.workunit.client.0.smithi129.stdout: "internal_metadata": 57793953 2024-03-26T00:35:04.791 INFO:tasks.workunit.client.0.smithi129.stdout: }, 2024-03-26T00:35:04.791 INFO:tasks.workunit.client.0.smithi129.stdout: "hb_peers": [], 2024-03-26T00:35:04.791 INFO:tasks.workunit.client.0.smithi129.stdout: "snap_trim_queue_len": 0, 2024-03-26T00:35:04.791 INFO:tasks.workunit.client.0.smithi129.stdout: "num_snap_trimming": 0, 2024-03-26T00:35:04.791 INFO:tasks.workunit.client.0.smithi129.stdout: "num_shards_repaired": 20, 2024-03-26T00:35:04.791 INFO:tasks.workunit.client.0.smithi129.stdout: "op_queue_age_hist": { 2024-03-26T00:35:04.791 INFO:tasks.workunit.client.0.smithi129.stdout: "histogram": [], 2024-03-26T00:35:04.791 INFO:tasks.workunit.client.0.smithi129.stdout: "upper_bound": 1 2024-03-26T00:35:04.791 INFO:tasks.workunit.client.0.smithi129.stdout: }, 2024-03-26T00:35:04.791 INFO:tasks.workunit.client.0.smithi129.stdout: "perf_stat": { 2024-03-26T00:35:04.792 INFO:tasks.workunit.client.0.smithi129.stdout: "commit_latency_ms": 0, 2024-03-26T00:35:04.792 INFO:tasks.workunit.client.0.smithi129.stdout: "apply_latency_ms": 0, 2024-03-26T00:35:04.792 INFO:tasks.workunit.client.0.smithi129.stdout: "commit_latency_ns": 0, 2024-03-26T00:35:04.792 INFO:tasks.workunit.client.0.smithi129.stdout: "apply_latency_ns": 0 2024-03-26T00:35:04.792 INFO:tasks.workunit.client.0.smithi129.stdout: }, 2024-03-26T00:35:04.792 INFO:tasks.workunit.client.0.smithi129.stdout: "alerts": [], 2024-03-26T00:35:04.792 INFO:tasks.workunit.client.0.smithi129.stdout: "network_ping_times": [] 2024-03-26T00:35:04.792 INFO:tasks.workunit.client.0.smithi129.stdout:} 2024-03-26T00:35:04.793 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-03-26T00:35:04.793 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-03-26T00:35:05.071 INFO:tasks.workunit.client.0.smithi129.stderr:dumped all 2024-03-26T00:35:05.089 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: COUNT=20 2024-03-26T00:35:05.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:720: TEST_repair_stats: test 20 = 20 2024-03-26T00:35:05.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-26T00:35:05.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-26T00:35:05.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-26T00:35:05.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-26T00:35:05.090 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:35:05.090 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:35:05.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:35:05.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:35:05.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:35:05.402 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:35:05.402 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-26T00:35:05.403 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-26T00:35:05.405 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-26T00:35:05.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-26T00:35:05.406 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-26T00:35:05.407 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-26T00:35:05.408 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:35:05.408 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-26T00:35:05.408 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-26T00:35:05.409 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:35:05.410 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-26T00:35:05.412 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-26T00:35:05.412 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-26T00:35:05.432 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-26T00:35:05.432 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:35:05.432 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:35:05.432 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-26T00:35:05.434 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-26T00:35:05.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-26T00:35:05.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-26T00:35:05.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-26T00:35:05.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-26T00:35:05.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-26T00:35:05.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-26T00:35:05.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-26T00:35:05.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-26T00:35:05.435 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:35:05.435 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:35:05.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:35:05.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:35:05.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:35:05.438 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:35:05.439 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-26T00:35:05.440 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-26T00:35:05.440 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-26T00:35:05.442 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-26T00:35:05.442 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-26T00:35:05.442 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-26T00:35:05.443 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:35:05.444 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-26T00:35:05.444 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-26T00:35:05.444 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:35:05.445 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-26T00:35:05.447 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-26T00:35:05.447 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-26T00:35:05.449 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-26T00:35:05.449 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:35:05.449 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:35:05.449 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-26T00:35:05.450 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-26T00:35:05.450 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-26T00:35:05.450 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-26T00:35:05.452 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-26T00:35:05.452 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:35:05.452 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:35:05.452 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61636 2024-03-26T00:35:05.456 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-26T00:35:05.456 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-26T00:35:05.456 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-26T00:35:05.456 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-26T00:35:05.456 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_repair_stats_ec td/osd-scrub-repair 2024-03-26T00:35:05.456 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:724: TEST_repair_stats_ec: local dir=td/osd-scrub-repair 2024-03-26T00:35:05.456 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:725: TEST_repair_stats_ec: local poolname=testpool 2024-03-26T00:35:05.456 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:726: TEST_repair_stats_ec: local OSDS=3 2024-03-26T00:35:05.456 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:727: TEST_repair_stats_ec: local OBJS=30 2024-03-26T00:35:05.456 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:729: TEST_repair_stats_ec: local REPAIRS=26 2024-03-26T00:35:05.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:730: TEST_repair_stats_ec: local allow_overwrites=false 2024-03-26T00:35:05.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:733: TEST_repair_stats_ec: run_mon td/osd-scrub-repair a 2024-03-26T00:35:05.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-26T00:35:05.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-26T00:35:05.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-26T00:35:05.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-26T00:35:05.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-26T00:35:05.457 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-26T00:35:05.995 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-26T00:35:05.995 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:35:05.995 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:35:05.996 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:35:05.996 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:35:05.996 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:35:05.996 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:35:05.996 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-26T00:35:06.247 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-26T00:35:06.249 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-26T00:35:06.249 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-26T00:35:06.249 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-26T00:35:06.250 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-26T00:35:06.250 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-26T00:35:06.252 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-26T00:35:06.252 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-26T00:35:06.252 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-26T00:35:06.252 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-26T00:35:06.252 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:35:06.252 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:35:06.252 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-26T00:35:06.253 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-26T00:35:06.253 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get fsid 2024-03-26T00:35:06.370 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-26T00:35:06.370 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-26T00:35:06.370 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-26T00:35:06.370 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-26T00:35:06.371 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-26T00:35:06.373 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-26T00:35:06.373 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-26T00:35:06.373 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-26T00:35:06.373 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-26T00:35:06.373 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:35:06.373 INFO:tasks.workunit.client.0.smithi129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:35:06.374 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61636/ceph-mon.a.asok 2024-03-26T00:35:06.374 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-26T00:35:06.374 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61636/ceph-mon.a.asok config get mon_host 2024-03-26T00:35:06.488 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:734: TEST_repair_stats_ec: run_mgr td/osd-scrub-repair x 2024-03-26T00:35:06.488 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-26T00:35:06.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-26T00:35:06.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-26T00:35:06.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-26T00:35:06.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-26T00:35:06.489 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-26T00:35:08.193 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-26T00:35:08.194 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:35:08.194 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:35:08.195 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:35:08.195 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:35:08.195 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:35:08.195 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:35:08.196 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-26T00:35:08.196 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-26T00:35:08.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats_ec: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-26T00:35:08.232 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: expr 3 - 1 2024-03-26T00:35:08.234 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: seq 0 2 2024-03-26T00:35:08.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-03-26T00:35:08.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-26T00:35:08.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-26T00:35:08.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-26T00:35:08.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-26T00:35:08.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-26T00:35:08.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:35:08.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:35:08.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:35:08.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:35:08.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:35:08.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-26T00:35:08.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-26T00:35:08.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-26T00:35:08.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-26T00:35:08.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:35:08.237 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-26T00:35:08.238 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:35:08.238 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:35:08.238 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:35:08.238 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:35:08.238 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:35:08.238 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:35:08.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:35:08.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:35:08.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-26T00:35:08.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-26T00:35:08.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:35:08.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:35:08.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:35:08.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:35:08.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:35:08.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:35:08.241 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-26T00:35:08.241 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-26T00:35:08.241 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-26T00:35:08.241 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-26T00:35:08.241 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=543d7e79-dd02-4143-b238-c09e1a037342 2024-03-26T00:35:08.241 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 543d7e79-dd02-4143-b238-c09e1a037342' 2024-03-26T00:35:08.241 INFO:tasks.workunit.client.0.smithi129.stdout:add osd0 543d7e79-dd02-4143-b238-c09e1a037342 2024-03-26T00:35:08.241 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-26T00:35:08.258 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA8GAJmJ5JZDxAAXZX3SrkYMkwTPEyOWxpb+g== 2024-03-26T00:35:08.258 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA8GAJmJ5JZDxAAXZX3SrkYMkwTPEyOWxpb+g=="}' 2024-03-26T00:35:08.258 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 543d7e79-dd02-4143-b238-c09e1a037342 -i td/osd-scrub-repair/0/new.json 2024-03-26T00:35:08.551 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-26T00:35:08.569 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-26T00:35:08.571 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQA8GAJmJ5JZDxAAXZX3SrkYMkwTPEyOWxpb+g== --osd-uuid 543d7e79-dd02-4143-b238-c09e1a037342 2024-03-26T00:35:08.605 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:35:08.599+0000 7f7c0e8127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:35:08.635 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:35:08.631+0000 7f7c0e8127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:35:08.637 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:35:08.631+0000 7f7c0e8127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:35:08.637 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:35:08.635+0000 7f7c0e8127c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-26T00:35:08.637 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:35:08.635+0000 7f7c0e8127c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-26T00:35:08.637 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:35:08.635+0000 7f7c0e8127c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-26T00:35:08.735 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:35:08.731+0000 7f7c0e8127c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-26T00:35:16.763 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-26T00:35:16.763 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-26T00:35:16.764 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd0 key to auth repository 2024-03-26T00:35:16.764 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-26T00:35:16.764 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-26T00:35:17.155 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-26T00:35:17.156 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.0 2024-03-26T00:35:17.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-26T00:35:17.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-26T00:35:17.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-26T00:35:17.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-26T00:35:17.189 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:35:17.183+0000 7feb58a907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:35:17.233 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:35:17.227+0000 7feb58a907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:35:17.234 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:35:17.231+0000 7feb58a907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:35:17.466 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-26T00:35:17.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:35:17.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-26T00:35:17.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:35:17.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:35:17.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:35:17.467 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:35:17.467 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-26T00:35:17.468 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:35:17.468 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:35:17.792 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:35:17.787+0000 7feb58a907c0 -1 Falling back to public interface 2024-03-26T00:35:17.818 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:35:18.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:35:18.820 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:35:18.820 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-26T00:35:18.821 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:35:18.821 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:35:18.821 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:35:19.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:35:20.141 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:35:20.141 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:35:20.142 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-26T00:35:20.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:35:20.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:35:20.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:35:20.462 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:35:20.726 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:35:20.723+0000 7feb58a907c0 -1 osd.0 0 log_to_monitors true 2024-03-26T00:35:21.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:35:21.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:35:21.464 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-26T00:35:21.465 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:35:21.465 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:35:21.465 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:35:21.810 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:35:22.811 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:35:22.811 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:35:22.811 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-26T00:35:22.812 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-26T00:35:22.812 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:35:22.812 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:35:23.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:35:24.114 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:35:24.114 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:35:24.115 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-26T00:35:24.115 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-26T00:35:24.115 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:35:24.115 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:35:24.436 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:35:25.437 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:35:25.437 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:35:25.438 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-26T00:35:25.438 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-26T00:35:25.438 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:35:25.438 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:35:25.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:35:26.992 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:35:26.993 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:35:26.993 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-26T00:35:26.993 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-26T00:35:26.993 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:35:26.993 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:35:27.315 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:35:28.316 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:35:28.317 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:35:28.317 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-26T00:35:28.317 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-26T00:35:28.317 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:35:28.318 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:35:28.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:35:29.715 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:35:29.715 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:35:29.715 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-26T00:35:29.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-26T00:35:29.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:35:29.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:35:30.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:35:31.184 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:35:31.184 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:35:31.184 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-26T00:35:31.185 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-26T00:35:31.185 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:35:31.185 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:35:31.494 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:35:32.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:35:32.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:35:32.497 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-26T00:35:32.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-26T00:35:32.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:35:32.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:35:33.106 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:35:34.108 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:35:34.108 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:35:34.108 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-26T00:35:34.109 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-26T00:35:34.109 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:35:34.109 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:35:34.430 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:35:35.432 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:35:35.432 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:35:35.432 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-26T00:35:35.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-26T00:35:35.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:35:35.433 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:35:35.901 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:35:36.903 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:35:36.903 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:35:36.903 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-26T00:35:36.903 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-26T00:35:36.904 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:35:36.904 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:35:37.313 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:35:38.315 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:35:38.315 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:35:38.315 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-26T00:35:38.316 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-26T00:35:38.316 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:35:38.316 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:35:38.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:35:39.699 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:35:39.700 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:35:39.700 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-26T00:35:39.700 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-26T00:35:39.701 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:35:39.701 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:35:40.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:35:41.184 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:35:41.184 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:35:41.184 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-26T00:35:41.185 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-26T00:35:41.185 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:35:41.185 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:35:42.025 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:35:43.026 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:35:43.026 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:35:43.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-26T00:35:43.027 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-26T00:35:43.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:35:43.027 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:35:44.091 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:35:45.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:35:45.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:35:45.094 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-26T00:35:45.094 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-26T00:35:45.094 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:35:45.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:35:45.827 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:35:46.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:35:46.829 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:35:46.830 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-26T00:35:46.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-26T00:35:46.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:35:46.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:35:47.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:35:48.993 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:35:48.993 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:35:48.994 INFO:tasks.workunit.client.0.smithi129.stdout:21 2024-03-26T00:35:48.994 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-26T00:35:48.994 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:35:48.994 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:35:49.759 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:35:50.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:35:50.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:35:50.761 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-26T00:35:50.762 INFO:tasks.workunit.client.0.smithi129.stdout:22 2024-03-26T00:35:50.762 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:35:50.763 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:35:51.425 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:35:52.427 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:35:52.427 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:35:52.427 INFO:tasks.workunit.client.0.smithi129.stdout:23 2024-03-26T00:35:52.428 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-03-26T00:35:52.428 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:35:52.428 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:35:53.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:35:54.080 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:35:54.080 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:35:54.080 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-03-26T00:35:54.081 INFO:tasks.workunit.client.0.smithi129.stdout:24 2024-03-26T00:35:54.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:35:54.081 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:35:54.382 INFO:tasks.workunit.client.0.smithi129.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/258610058,v1:127.0.0.1:6803/258610058] [v2:127.0.0.1:6804/258610058,v1:127.0.0.1:6805/258610058] exists,up 543d7e79-dd02-4143-b238-c09e1a037342 2024-03-26T00:35:54.384 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:35:54.384 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:35:54.384 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:35:54.384 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-03-26T00:35:54.385 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-26T00:35:54.385 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-26T00:35:54.385 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-26T00:35:54.385 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-26T00:35:54.385 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-26T00:35:54.385 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:35:54.385 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:35:54.385 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:35:54.385 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:35:54.385 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:35:54.385 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:35:54.385 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:35:54.385 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-26T00:35:54.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-26T00:35:54.386 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:35:54.386 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-26T00:35:54.386 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:35:54.386 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:35:54.388 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:35:54.388 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:35:54.388 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:35:54.388 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:35:54.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:35:54.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:35:54.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-26T00:35:54.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-26T00:35:54.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:35:54.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:35:54.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:35:54.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:35:54.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:35:54.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:35:54.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-26T00:35:54.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-26T00:35:54.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:35:54.389 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-26T00:35:54.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=add161d5-0034-4ce0-a147-8d72f0847aa5 2024-03-26T00:35:54.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 add161d5-0034-4ce0-a147-8d72f0847aa5' 2024-03-26T00:35:54.390 INFO:tasks.workunit.client.0.smithi129.stdout:add osd1 add161d5-0034-4ce0-a147-8d72f0847aa5 2024-03-26T00:35:54.390 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-26T00:35:54.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBqGAJmh76TGBAAGkqgqXM9FL9yKwOb3qhWYQ== 2024-03-26T00:35:54.413 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBqGAJmh76TGBAAGkqgqXM9FL9yKwOb3qhWYQ=="}' 2024-03-26T00:35:54.414 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new add161d5-0034-4ce0-a147-8d72f0847aa5 -i td/osd-scrub-repair/1/new.json 2024-03-26T00:35:54.812 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-26T00:35:54.832 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-26T00:35:54.834 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBqGAJmh76TGBAAGkqgqXM9FL9yKwOb3qhWYQ== --osd-uuid add161d5-0034-4ce0-a147-8d72f0847aa5 2024-03-26T00:35:54.868 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:35:54.862+0000 7f181646b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:35:54.956 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:35:54.950+0000 7f181646b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:35:54.958 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:35:54.954+0000 7f181646b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:35:54.958 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:35:54.954+0000 7f181646b7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-26T00:35:54.958 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:35:54.954+0000 7f181646b7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-26T00:35:54.958 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:35:54.954+0000 7f181646b7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-26T00:35:55.044 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:35:55.042+0000 7f181646b7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-26T00:36:03.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-26T00:36:03.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-26T00:36:03.589 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd1 key to auth repository 2024-03-26T00:36:03.589 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-26T00:36:03.590 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-26T00:36:03.976 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.1 2024-03-26T00:36:03.977 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-26T00:36:03.977 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-26T00:36:03.977 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-26T00:36:03.978 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-26T00:36:03.978 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-26T00:36:04.010 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:36:04.006+0000 7f22fb59d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:36:04.061 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:36:04.054+0000 7f22fb59d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:36:04.062 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:36:04.058+0000 7f22fb59d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:36:04.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-26T00:36:04.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:36:04.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:36:04.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:36:04.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:36:04.280 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:36:04.280 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-26T00:36:04.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:36:04.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:36:04.281 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:36:04.579 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:36:04.655 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:36:04.650+0000 7f22fb59d7c0 -1 Falling back to public interface 2024-03-26T00:36:05.580 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:36:05.581 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:36:05.581 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-26T00:36:05.582 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:36:05.582 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:36:05.582 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:36:05.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:36:06.886 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:36:06.886 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:36:06.887 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-26T00:36:06.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:36:06.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:36:06.887 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:36:07.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:36:07.691 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:36:07.686+0000 7f22fb59d7c0 -1 osd.1 0 log_to_monitors true 2024-03-26T00:36:08.206 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:36:08.206 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:36:08.206 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-26T00:36:08.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:36:08.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:36:08.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:36:08.713 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:36:09.714 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:36:09.715 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:36:09.715 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-26T00:36:09.715 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-26T00:36:09.715 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:36:09.716 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:36:10.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:36:11.141 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:36:11.141 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:36:11.142 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-26T00:36:11.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-26T00:36:11.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:36:11.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:36:11.668 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:36:12.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:36:12.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:36:12.670 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-26T00:36:12.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-26T00:36:12.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:36:12.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:36:13.053 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:36:14.054 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:36:14.054 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:36:14.055 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-26T00:36:14.055 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-26T00:36:14.055 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:36:14.055 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:36:14.373 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:36:15.375 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:36:15.375 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:36:15.375 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-26T00:36:15.376 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-26T00:36:15.376 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:36:15.376 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:36:15.697 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:36:16.699 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:36:16.699 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:36:16.699 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-26T00:36:16.700 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-26T00:36:16.700 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:36:16.700 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:36:17.099 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:36:18.101 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:36:18.101 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:36:18.102 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-26T00:36:18.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-26T00:36:18.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:36:18.102 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:36:18.409 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:36:19.411 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:36:19.411 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:36:19.411 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-26T00:36:19.412 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-26T00:36:19.412 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:36:19.412 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:36:19.889 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:36:20.891 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:36:20.891 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:36:20.892 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-26T00:36:20.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-26T00:36:20.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:36:20.892 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:36:21.286 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:36:22.287 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:36:22.287 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:36:22.288 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-26T00:36:22.288 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-26T00:36:22.288 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:36:22.288 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:36:22.597 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:36:23.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:36:23.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:36:23.599 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-26T00:36:23.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-26T00:36:23.600 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:36:23.600 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:36:24.063 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:36:25.064 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:36:25.065 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:36:25.065 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-26T00:36:25.065 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-26T00:36:25.065 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:36:25.065 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:36:25.388 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:36:26.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:36:26.391 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:36:26.391 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-26T00:36:26.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-26T00:36:26.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:36:26.392 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:36:26.882 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:36:27.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:36:27.884 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:36:27.884 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-26T00:36:27.885 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-26T00:36:27.885 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:36:27.885 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:36:28.183 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:36:29.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:36:29.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:36:29.187 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-26T00:36:29.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-26T00:36:29.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:36:29.188 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:36:29.493 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:36:30.495 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:36:30.495 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:36:30.495 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-26T00:36:30.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-26T00:36:30.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:36:30.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:36:31.229 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:36:32.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:36:32.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:36:32.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-26T00:36:32.231 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-26T00:36:32.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:36:32.232 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:36:32.916 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:36:33.918 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:36:33.918 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:36:33.918 INFO:tasks.workunit.client.0.smithi129.stdout:21 2024-03-26T00:36:33.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-26T00:36:33.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:36:33.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:36:34.729 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:36:35.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:36:35.731 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:36:35.732 INFO:tasks.workunit.client.0.smithi129.stdout:22 2024-03-26T00:36:35.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-26T00:36:35.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:36:35.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:36:36.496 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:36:37.497 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:36:37.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:36:37.498 INFO:tasks.workunit.client.0.smithi129.stdout:23 2024-03-26T00:36:37.498 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-03-26T00:36:37.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:36:37.499 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:36:38.171 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:36:39.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:36:39.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:36:39.173 INFO:tasks.workunit.client.0.smithi129.stdout:24 2024-03-26T00:36:39.174 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-03-26T00:36:39.174 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:36:39.174 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:36:39.485 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:36:40.487 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:36:40.487 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:36:40.487 INFO:tasks.workunit.client.0.smithi129.stdout:25 2024-03-26T00:36:40.488 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-03-26T00:36:40.488 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:36:40.488 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:36:41.001 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:36:42.003 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:36:42.003 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:36:42.004 INFO:tasks.workunit.client.0.smithi129.stdout:26 2024-03-26T00:36:42.004 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 26 2024-03-26T00:36:42.004 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:36:42.005 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:36:42.321 INFO:tasks.workunit.client.0.smithi129.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/8226371,v1:127.0.0.1:6811/8226371] [v2:127.0.0.1:6812/8226371,v1:127.0.0.1:6813/8226371] exists,up add161d5-0034-4ce0-a147-8d72f0847aa5 2024-03-26T00:36:42.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:36:42.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:36:42.329 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:36:42.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-03-26T00:36:42.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 2 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-26T00:36:42.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-26T00:36:42.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-26T00:36:42.330 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-26T00:36:42.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-26T00:36:42.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-26T00:36:42.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:36:42.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:36:42.331 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:36:42.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:36:42.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-26T00:36:42.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-26T00:36:42.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-26T00:36:42.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-26T00:36:42.332 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:36:42.333 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-26T00:36:42.333 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:36:42.333 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:36:42.333 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:36:42.333 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:36:42.334 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:36:42.334 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:36:42.334 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:36:42.334 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:36:42.334 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-26T00:36:42.335 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-26T00:36:42.335 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:36:42.335 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:36:42.335 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:36:42.335 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:36:42.335 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:36:42.336 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:36:42.336 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-26T00:36:42.336 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-26T00:36:42.336 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-26T00:36:42.336 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-26T00:36:42.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fcba7f0d-e4d8-4624-bb69-08472ceff17a 2024-03-26T00:36:42.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 fcba7f0d-e4d8-4624-bb69-08472ceff17a' 2024-03-26T00:36:42.337 INFO:tasks.workunit.client.0.smithi129.stdout:add osd2 fcba7f0d-e4d8-4624-bb69-08472ceff17a 2024-03-26T00:36:42.338 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-26T00:36:42.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCaGAJmjWnDFBAAQMOj5XXf0/ojyXFy0eFLCg== 2024-03-26T00:36:42.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCaGAJmjWnDFBAAQMOj5XXf0/ojyXFy0eFLCg=="}' 2024-03-26T00:36:42.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fcba7f0d-e4d8-4624-bb69-08472ceff17a -i td/osd-scrub-repair/2/new.json 2024-03-26T00:36:42.758 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-26T00:36:42.779 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-26T00:36:42.781 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCaGAJmjWnDFBAAQMOj5XXf0/ojyXFy0eFLCg== --osd-uuid fcba7f0d-e4d8-4624-bb69-08472ceff17a 2024-03-26T00:36:42.818 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:36:42.817+0000 7f85500cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:36:42.894 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:36:42.893+0000 7f85500cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:36:42.896 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:36:42.893+0000 7f85500cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:36:42.896 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:36:42.893+0000 7f85500cc7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-26T00:36:42.896 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:36:42.893+0000 7f85500cc7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-26T00:36:42.896 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:36:42.893+0000 7f85500cc7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-26T00:36:43.012 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:36:43.009+0000 7f85500cc7c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-26T00:36:51.588 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-26T00:36:51.589 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-26T00:36:51.590 INFO:tasks.workunit.client.0.smithi129.stdout:adding osd2 key to auth repository 2024-03-26T00:36:51.590 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-26T00:36:51.590 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-26T00:36:51.973 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.2 2024-03-26T00:36:51.974 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-26T00:36:51.974 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-26T00:36:51.974 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-26T00:36:51.974 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-26T00:36:51.974 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-26T00:36:52.002 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:36:52.001+0000 7fd0cd00b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:36:52.004 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:36:52.001+0000 7fd0cd00b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:36:52.005 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:36:52.005+0000 7fd0cd00b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:36:52.290 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-26T00:36:52.291 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:36:52.291 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-26T00:36:52.291 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:36:52.291 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:36:52.291 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:36:52.291 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-26T00:36:52.292 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:36:52.292 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:36:52.292 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:36:52.539 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:36:52.537+0000 7fd0cd00b7c0 -1 Falling back to public interface 2024-03-26T00:36:52.604 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:36:53.606 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:36:53.606 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:36:53.606 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-26T00:36:53.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:36:53.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:36:53.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:36:53.919 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:36:54.921 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:36:54.921 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:36:54.921 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-26T00:36:54.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:36:54.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:36:54.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:36:55.239 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:36:55.531 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:36:55.529+0000 7fd0cd00b7c0 -1 osd.2 0 log_to_monitors true 2024-03-26T00:36:56.241 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:36:56.242 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:36:56.242 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:36:56.242 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-26T00:36:56.242 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:36:56.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:36:56.734 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:36:57.736 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:36:57.736 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:36:57.736 INFO:tasks.workunit.client.0.smithi129.stdout:4 2024-03-26T00:36:57.737 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-26T00:36:57.737 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:36:57.737 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:36:58.195 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:36:59.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:36:59.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:36:59.198 INFO:tasks.workunit.client.0.smithi129.stdout:5 2024-03-26T00:36:59.198 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-26T00:36:59.198 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:36:59.198 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:36:59.668 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:37:00.672 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:37:00.672 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:37:00.673 INFO:tasks.workunit.client.0.smithi129.stdout:6 2024-03-26T00:37:00.673 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-26T00:37:00.673 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:37:00.673 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:37:00.994 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:37:01.996 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:37:01.996 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:37:01.996 INFO:tasks.workunit.client.0.smithi129.stdout:7 2024-03-26T00:37:01.997 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-26T00:37:01.997 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:37:01.997 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:37:02.305 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:37:03.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:37:03.307 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:37:03.307 INFO:tasks.workunit.client.0.smithi129.stdout:8 2024-03-26T00:37:03.308 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-26T00:37:03.308 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:37:03.308 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:37:03.740 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:37:04.741 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:37:04.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:37:04.742 INFO:tasks.workunit.client.0.smithi129.stdout:9 2024-03-26T00:37:04.742 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-26T00:37:04.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:37:04.743 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:37:05.173 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:37:06.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:37:06.175 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:37:06.176 INFO:tasks.workunit.client.0.smithi129.stdout:10 2024-03-26T00:37:06.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-26T00:37:06.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:37:06.176 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:37:06.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:37:07.483 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:37:07.483 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:37:07.483 INFO:tasks.workunit.client.0.smithi129.stdout:11 2024-03-26T00:37:07.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-26T00:37:07.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:37:07.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:37:07.948 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:37:08.950 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:37:08.950 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:37:08.951 INFO:tasks.workunit.client.0.smithi129.stdout:12 2024-03-26T00:37:08.951 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-26T00:37:08.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:37:08.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:37:09.271 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:37:10.273 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:37:10.273 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:37:10.273 INFO:tasks.workunit.client.0.smithi129.stdout:13 2024-03-26T00:37:10.274 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-26T00:37:10.274 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:37:10.274 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:37:10.750 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:37:11.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:37:11.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:37:11.753 INFO:tasks.workunit.client.0.smithi129.stdout:14 2024-03-26T00:37:11.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-26T00:37:11.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:37:11.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:37:12.065 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:37:13.066 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:37:13.066 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:37:13.067 INFO:tasks.workunit.client.0.smithi129.stdout:15 2024-03-26T00:37:13.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-26T00:37:13.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:37:13.067 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:37:13.405 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:37:14.407 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:37:14.407 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:37:14.407 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-26T00:37:14.408 INFO:tasks.workunit.client.0.smithi129.stdout:16 2024-03-26T00:37:14.408 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:37:14.408 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:37:15.213 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:37:16.215 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:37:16.215 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:37:16.215 INFO:tasks.workunit.client.0.smithi129.stdout:17 2024-03-26T00:37:16.216 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-26T00:37:16.216 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:37:16.216 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:37:16.787 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:37:17.790 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:37:17.790 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:37:17.790 INFO:tasks.workunit.client.0.smithi129.stdout:18 2024-03-26T00:37:17.791 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-26T00:37:17.791 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:37:17.791 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:37:18.460 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:37:19.462 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:37:19.463 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:37:19.463 INFO:tasks.workunit.client.0.smithi129.stdout:19 2024-03-26T00:37:19.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-26T00:37:19.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:37:19.464 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:37:19.905 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:37:20.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:37:20.907 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:37:20.907 INFO:tasks.workunit.client.0.smithi129.stdout:20 2024-03-26T00:37:20.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-26T00:37:20.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:37:20.908 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:37:21.971 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:37:22.973 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:37:22.974 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:37:22.974 INFO:tasks.workunit.client.0.smithi129.stdout:21 2024-03-26T00:37:22.974 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-26T00:37:22.975 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:37:22.975 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:37:23.952 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:37:24.954 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:37:24.954 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:37:24.954 INFO:tasks.workunit.client.0.smithi129.stdout:22 2024-03-26T00:37:24.955 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-26T00:37:24.955 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:37:24.955 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:37:25.417 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:37:26.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:37:26.419 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:37:26.419 INFO:tasks.workunit.client.0.smithi129.stdout:23 2024-03-26T00:37:26.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-03-26T00:37:26.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:37:26.420 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:37:26.873 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:37:27.875 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:37:27.875 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:37:27.875 INFO:tasks.workunit.client.0.smithi129.stdout:24 2024-03-26T00:37:27.876 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-03-26T00:37:27.876 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:37:27.876 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:37:28.237 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:37:29.238 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:37:29.239 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:37:29.239 INFO:tasks.workunit.client.0.smithi129.stdout:25 2024-03-26T00:37:29.239 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-03-26T00:37:29.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:37:29.240 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-26T00:37:29.637 INFO:tasks.workunit.client.0.smithi129.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3488549365,v1:127.0.0.1:6819/3488549365] [v2:127.0.0.1:6820/3488549365,v1:127.0.0.1:6821/3488549365] exists,up fcba7f0d-e4d8-4624-bb69-08472ceff17a 2024-03-26T00:37:29.638 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:37:29.638 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:37:29.638 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:37:29.638 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:742: TEST_repair_stats_ec: create_ec_pool testpool false k=2 m=1 2024-03-26T00:37:29.638 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=testpool 2024-03-26T00:37:29.638 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-26T00:37:29.638 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-03-26T00:37:29.639 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-26T00:37:29.639 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-03-26T00:37:30.126 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool testpool 1 1 erasure myprofile 2024-03-26T00:37:30.126 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 erasure myprofile 2024-03-26T00:37:31.073 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'testpool' created 2024-03-26T00:37:31.093 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-26T00:37:32.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-03-26T00:37:32.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-26T00:37:32.095 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:37:32.096 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:37:32.096 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:37:32.096 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:37:32.097 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:37:32.097 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:37:32.098 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:37:32.098 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:37:32.098 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:37:32.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:37:32.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:37:32.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:37:32.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:37:32.236 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:37:32.236 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:37:32.575 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:37:32.575 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:37:32.575 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-26T00:37:32.575 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:37:32.576 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:37:32.576 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:37:32.723 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836505 2024-03-26T00:37:32.724 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836505 2024-03-26T00:37:32.724 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505' 2024-03-26T00:37:32.724 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:37:32.724 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:37:32.937 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672983 2024-03-26T00:37:32.937 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672983 2024-03-26T00:37:32.937 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672983' 2024-03-26T00:37:32.937 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:37:32.937 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:37:33.087 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509444 2024-03-26T00:37:33.087 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509444 2024-03-26T00:37:33.087 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672983 2-64424509444' 2024-03-26T00:37:33.087 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:37:33.087 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836505 2024-03-26T00:37:33.087 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:37:33.088 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:37:33.089 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836505 2024-03-26T00:37:33.089 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:37:33.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836505 2024-03-26T00:37:33.090 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-03-26T00:37:33.091 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 21474836505 2024-03-26T00:37:33.091 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:37:33.479 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836505 -lt 21474836505 2024-03-26T00:37:33.480 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:37:33.480 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672983 2024-03-26T00:37:33.480 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:37:33.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:37:33.482 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672983 2024-03-26T00:37:33.482 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:37:33.483 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672983 2024-03-26T00:37:33.484 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-03-26T00:37:33.484 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 42949672983 2024-03-26T00:37:33.484 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:37:33.876 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672982 -lt 42949672983 2024-03-26T00:37:33.876 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:37:34.878 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:37:34.879 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:37:35.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672983 -lt 42949672983 2024-03-26T00:37:35.197 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:37:35.198 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509444 2024-03-26T00:37:35.198 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:37:35.200 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:37:35.201 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:37:35.201 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509444 2024-03-26T00:37:35.202 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509444 2024-03-26T00:37:35.202 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 64424509444 2024-03-26T00:37:35.203 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2024-03-26T00:37:35.203 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:37:35.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509445 -lt 64424509444 2024-03-26T00:37:35.600 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:37:35.601 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:37:35.601 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:37:35.975 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-26T00:37:35.975 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:37:35.976 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:37:35.976 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:37:35.976 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:37:35.976 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:37:35.977 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:37:35.977 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:37:36.271 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-26T00:37:36.271 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:37:36.272 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:37:36.272 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:37:36.643 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-03-26T00:37:36.643 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-03-26T00:37:36.643 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-26T00:37:36.643 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-03-26T00:37:36.643 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-26T00:37:36.644 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-26T00:37:36.744 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-26T00:37:36.744 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:37:36.746 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:37:36.746 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:37:36.746 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:37:36.746 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:37:36.746 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:37:36.746 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:37:37.112 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-26T00:37:37.113 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:37:37.114 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:37:37.114 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:37:37.472 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-03-26T00:37:37.472 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-03-26T00:37:37.472 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-26T00:37:37.473 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-26T00:37:37.473 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-26T00:37:37.473 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-26T00:37:37.473 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-26T00:37:37.473 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-26T00:37:37.474 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-26T00:37:37.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-26T00:37:37.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-26T00:37:37.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-26T00:37:37.839 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-26T00:37:37.840 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-26T00:37:38.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-26T00:37:38.041 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:37:38.041 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:37:38.041 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:37:38.041 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:37:38.041 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:37:38.042 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:37:38.042 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:37:38.353 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-26T00:37:38.354 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:37:38.354 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:37:38.354 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:37:38.714 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-03-26T00:37:38.714 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-03-26T00:37:38.714 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-26T00:37:38.714 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-26T00:37:38.714 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-26T00:37:38.714 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-26T00:37:38.714 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-26T00:37:38.715 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-26T00:37:38.715 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-26T00:37:39.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-26T00:37:39.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-26T00:37:39.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-26T00:37:39.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-26T00:37:39.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-26T00:37:39.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-26T00:37:39.587 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:37:39.588 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:37:39.589 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:37:39.589 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:37:39.589 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:37:39.589 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:37:39.589 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:37:39.899 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-26T00:37:39.899 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:37:39.900 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:37:39.900 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:37:40.282 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-26T00:37:40.282 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:37:40.282 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:37:40.282 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-26T00:37:40.282 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:745: TEST_repair_stats_ec: local payload=ABCDEF 2024-03-26T00:37:40.282 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: echo ABCDEF 2024-03-26T00:37:40.283 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: seq 1 30 2024-03-26T00:37:40.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-26T00:37:40.284 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:37:40.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-26T00:37:40.513 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:37:40.798 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-26T00:37:40.798 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:37:41.140 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-26T00:37:41.141 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:37:41.416 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-26T00:37:41.416 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:37:41.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-26T00:37:41.752 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:37:42.053 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-26T00:37:42.053 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:37:42.371 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-26T00:37:42.372 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:37:42.714 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-26T00:37:42.715 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:37:43.024 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-26T00:37:43.024 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:37:43.412 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-26T00:37:43.412 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj11 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:37:43.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-26T00:37:43.711 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj12 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:37:44.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-26T00:37:44.078 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj13 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:37:44.363 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-26T00:37:44.363 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj14 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:37:44.673 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-26T00:37:44.673 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj15 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:37:45.016 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-26T00:37:45.016 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj16 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:37:45.317 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-26T00:37:45.317 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj17 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:37:45.695 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-26T00:37:45.695 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj18 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:37:46.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-26T00:37:46.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj19 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:37:46.389 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-26T00:37:46.390 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj20 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:37:46.666 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-26T00:37:46.666 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj21 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:37:46.941 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-26T00:37:46.941 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj22 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:37:47.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-26T00:37:47.276 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj23 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:37:47.569 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-26T00:37:47.569 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj24 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:37:47.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-26T00:37:47.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj25 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:37:48.213 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-26T00:37:48.213 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj26 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:37:48.540 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-26T00:37:48.540 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj27 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:37:48.858 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-26T00:37:48.858 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj28 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:37:49.161 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-26T00:37:49.162 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj29 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:37:49.503 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-26T00:37:49.503 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj30 td/osd-scrub-repair/ORIGINAL 2024-03-26T00:37:49.782 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:754: TEST_repair_stats_ec: get_not_primary testpool obj1 2024-03-26T00:37:49.782 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-26T00:37:49.782 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-03-26T00:37:49.783 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool obj1 2024-03-26T00:37:49.783 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-26T00:37:49.783 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-03-26T00:37:49.783 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-26T00:37:49.783 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-03-26T00:37:50.111 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-26T00:37:50.111 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool obj1 2024-03-26T00:37:50.111 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-26T00:37:50.473 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:754: TEST_repair_stats_ec: local other=0 2024-03-26T00:37:50.474 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:755: TEST_repair_stats_ec: get_pg testpool obj1 2024-03-26T00:37:50.474 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-03-26T00:37:50.474 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-03-26T00:37:50.474 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-03-26T00:37:50.475 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-26T00:37:50.792 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:755: TEST_repair_stats_ec: local pgid=1.0 2024-03-26T00:37:50.793 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:756: TEST_repair_stats_ec: get_primary testpool obj1 2024-03-26T00:37:50.793 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-26T00:37:50.793 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-03-26T00:37:50.794 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-26T00:37:50.794 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-03-26T00:37:51.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:756: TEST_repair_stats_ec: local primary=1 2024-03-26T00:37:51.194 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:758: TEST_repair_stats_ec: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-26T00:37:51.195 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:37:51.195 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:37:51.195 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:37:51.195 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:37:51.195 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:37:52.507 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:37:52.508 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:759: TEST_repair_stats_ec: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-26T00:37:52.509 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:37:52.509 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:37:52.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:37:52.509 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:37:52.510 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:37:52.824 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:37:52.825 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: seq 1 26 2024-03-26T00:37:52.827 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-26T00:37:52.828 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 1 % 2 2024-03-26T00:37:52.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-26T00:37:52.830 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 remove 2024-03-26T00:37:52.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:37:52.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:37:52.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:37:52.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:37:52.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:37:52.831 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 remove 2024-03-26T00:37:55.160 INFO:tasks.workunit.client.0.smithi129.stdout:remove 0#1:ff7b1f36:::obj1:head# 2024-03-26T00:37:55.704 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-26T00:37:55.705 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 2 % 2 2024-03-26T00:37:55.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-26T00:37:55.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-03-26T00:37:55.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:37:55.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:37:55.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:37:55.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:37:55.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:37:55.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-03-26T00:37:57.424 INFO:tasks.workunit.client.0.smithi129.stdout:remove 1#1:104778fc:::obj2:head# 2024-03-26T00:37:57.964 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-26T00:37:57.965 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 3 % 2 2024-03-26T00:37:57.966 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-26T00:37:57.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj3 remove 2024-03-26T00:37:57.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:37:57.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:37:57.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:37:57.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:37:57.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:37:57.967 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj3 remove 2024-03-26T00:37:59.152 INFO:tasks.workunit.client.0.smithi129.stdout:remove 0#1:8dd16f86:::obj3:head# 2024-03-26T00:37:59.687 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-26T00:37:59.688 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 4 % 2 2024-03-26T00:37:59.690 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-26T00:37:59.690 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj4 remove 2024-03-26T00:37:59.690 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:37:59.690 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:37:59.690 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:37:59.690 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:37:59.690 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:37:59.690 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj4 remove 2024-03-26T00:38:00.796 INFO:tasks.workunit.client.0.smithi129.stdout:remove 1#1:0ee9ae15:::obj4:head# 2024-03-26T00:38:01.334 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-26T00:38:01.335 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 5 % 2 2024-03-26T00:38:01.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-26T00:38:01.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj5 remove 2024-03-26T00:38:01.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:38:01.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:38:01.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:38:01.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:38:01.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:38:01.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj5 remove 2024-03-26T00:38:02.249 INFO:tasks.workunit.client.0.smithi129.stdout:remove 0#1:c52c9666:::obj5:head# 2024-03-26T00:38:02.786 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-26T00:38:02.787 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 6 % 2 2024-03-26T00:38:02.789 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-26T00:38:02.790 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj6 remove 2024-03-26T00:38:02.790 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:38:02.790 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:38:02.790 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:38:02.790 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:38:02.790 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:38:02.790 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj6 remove 2024-03-26T00:38:03.760 INFO:tasks.workunit.client.0.smithi129.stdout:remove 1#1:233a42c1:::obj6:head# 2024-03-26T00:38:04.312 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-26T00:38:04.313 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 7 % 2 2024-03-26T00:38:04.315 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-26T00:38:04.315 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj7 remove 2024-03-26T00:38:04.315 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:38:04.315 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:38:04.315 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:38:04.315 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:38:04.315 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:38:04.315 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj7 remove 2024-03-26T00:38:05.245 INFO:tasks.workunit.client.0.smithi129.stdout:remove 0#1:9a09113a:::obj7:head# 2024-03-26T00:38:05.780 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-26T00:38:05.781 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 8 % 2 2024-03-26T00:38:05.783 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-26T00:38:05.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj8 remove 2024-03-26T00:38:05.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:38:05.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:38:05.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:38:05.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:38:05.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:38:05.784 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj8 remove 2024-03-26T00:38:06.681 INFO:tasks.workunit.client.0.smithi129.stdout:remove 1#1:a557efb1:::obj8:head# 2024-03-26T00:38:07.216 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-26T00:38:07.217 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 9 % 2 2024-03-26T00:38:07.218 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-26T00:38:07.219 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj9 remove 2024-03-26T00:38:07.219 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:38:07.219 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:38:07.219 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:38:07.219 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:38:07.219 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:38:07.219 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj9 remove 2024-03-26T00:38:08.133 INFO:tasks.workunit.client.0.smithi129.stdout:remove 0#1:fe351e27:::obj9:head# 2024-03-26T00:38:08.666 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-26T00:38:08.667 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 10 % 2 2024-03-26T00:38:08.669 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-26T00:38:08.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj10 remove 2024-03-26T00:38:08.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:38:08.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:38:08.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:38:08.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:38:08.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:38:08.670 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj10 remove 2024-03-26T00:38:09.669 INFO:tasks.workunit.client.0.smithi129.stdout:remove 1#1:d1337354:::obj10:head# 2024-03-26T00:38:10.204 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-26T00:38:10.204 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 11 % 2 2024-03-26T00:38:10.206 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-26T00:38:10.206 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj11 remove 2024-03-26T00:38:10.206 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:38:10.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:38:10.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:38:10.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:38:10.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:38:10.207 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj11 remove 2024-03-26T00:38:11.121 INFO:tasks.workunit.client.0.smithi129.stdout:remove 0#1:eeae2a94:::obj11:head# 2024-03-26T00:38:12.165 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-26T00:38:12.165 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 12 % 2 2024-03-26T00:38:12.167 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-26T00:38:12.167 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj12 remove 2024-03-26T00:38:12.167 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:38:12.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:38:12.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:38:12.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:38:12.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:38:12.168 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj12 remove 2024-03-26T00:38:13.116 INFO:tasks.workunit.client.0.smithi129.stdout:remove 1#1:af5f95cb:::obj12:head# 2024-03-26T00:38:14.152 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-26T00:38:14.153 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 13 % 2 2024-03-26T00:38:14.156 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-26T00:38:14.156 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj13 remove 2024-03-26T00:38:14.156 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:38:14.156 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:38:14.156 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:38:14.156 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:38:14.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:38:14.157 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj13 remove 2024-03-26T00:38:15.070 INFO:tasks.workunit.client.0.smithi129.stdout:remove 0#1:a61ad63e:::obj13:head# 2024-03-26T00:38:15.615 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-26T00:38:15.616 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 14 % 2 2024-03-26T00:38:15.618 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-26T00:38:15.619 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj14 remove 2024-03-26T00:38:15.619 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:38:15.619 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:38:15.619 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:38:15.619 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:38:15.619 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:38:15.619 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj14 remove 2024-03-26T00:38:16.513 INFO:tasks.workunit.client.0.smithi129.stdout:remove 1#1:e0a44829:::obj14:head# 2024-03-26T00:38:17.047 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-26T00:38:17.048 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 15 % 2 2024-03-26T00:38:17.050 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-26T00:38:17.051 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj15 remove 2024-03-26T00:38:17.051 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:38:17.051 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:38:17.051 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:38:17.051 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:38:17.051 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:38:17.051 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj15 remove 2024-03-26T00:38:17.991 INFO:tasks.workunit.client.0.smithi129.stdout:remove 0#1:ab946124:::obj15:head# 2024-03-26T00:38:18.526 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-26T00:38:18.526 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 16 % 2 2024-03-26T00:38:18.529 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-26T00:38:18.529 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj16 remove 2024-03-26T00:38:18.529 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:38:18.530 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:38:18.530 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:38:18.530 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:38:18.530 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:38:18.530 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj16 remove 2024-03-26T00:38:19.499 INFO:tasks.workunit.client.0.smithi129.stdout:remove 1#1:461f8b5e:::obj16:head# 2024-03-26T00:38:20.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-26T00:38:20.041 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 17 % 2 2024-03-26T00:38:20.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-26T00:38:20.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj17 remove 2024-03-26T00:38:20.044 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:38:20.044 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:38:20.044 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:38:20.044 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:38:20.044 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:38:20.044 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj17 remove 2024-03-26T00:38:20.914 INFO:tasks.workunit.client.0.smithi129.stdout:remove 0#1:24fc6e92:::obj17:head# 2024-03-26T00:38:21.478 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-26T00:38:21.479 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 18 % 2 2024-03-26T00:38:21.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-26T00:38:21.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj18 remove 2024-03-26T00:38:21.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:38:21.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:38:21.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:38:21.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:38:21.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:38:21.481 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj18 remove 2024-03-26T00:38:22.388 INFO:tasks.workunit.client.0.smithi129.stdout:remove 1#1:b9836a99:::obj18:head# 2024-03-26T00:38:22.927 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-26T00:38:22.928 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 19 % 2 2024-03-26T00:38:22.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-26T00:38:22.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj19 remove 2024-03-26T00:38:22.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:38:22.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:38:22.930 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:38:22.931 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:38:22.931 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:38:22.931 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj19 remove 2024-03-26T00:38:23.890 INFO:tasks.workunit.client.0.smithi129.stdout:remove 0#1:00b31b6c:::obj19:head# 2024-03-26T00:38:24.432 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-26T00:38:24.432 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 20 % 2 2024-03-26T00:38:24.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-26T00:38:24.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj20 remove 2024-03-26T00:38:24.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:38:24.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:38:24.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:38:24.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:38:24.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:38:24.435 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj20 remove 2024-03-26T00:38:25.441 INFO:tasks.workunit.client.0.smithi129.stdout:remove 1#1:24b5611b:::obj20:head# 2024-03-26T00:38:25.988 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-26T00:38:25.988 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 21 % 2 2024-03-26T00:38:25.990 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-26T00:38:25.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj21 remove 2024-03-26T00:38:25.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:38:25.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:38:25.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:38:25.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:38:25.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:38:25.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj21 remove 2024-03-26T00:38:26.935 INFO:tasks.workunit.client.0.smithi129.stdout:remove 0#1:1243fe1a:::obj21:head# 2024-03-26T00:38:27.471 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-26T00:38:27.472 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 22 % 2 2024-03-26T00:38:27.474 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-26T00:38:27.474 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj22 remove 2024-03-26T00:38:27.474 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:38:27.474 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:38:27.474 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:38:27.474 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:38:27.474 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:38:27.474 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj22 remove 2024-03-26T00:38:28.431 INFO:tasks.workunit.client.0.smithi129.stdout:remove 1#1:c0c49def:::obj22:head# 2024-03-26T00:38:28.968 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-26T00:38:28.969 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 23 % 2 2024-03-26T00:38:28.971 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-26T00:38:28.971 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj23 remove 2024-03-26T00:38:28.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:38:28.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:38:28.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:38:28.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:38:28.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:38:28.972 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj23 remove 2024-03-26T00:38:30.058 INFO:tasks.workunit.client.0.smithi129.stdout:remove 0#1:242c40e0:::obj23:head# 2024-03-26T00:38:30.596 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-26T00:38:30.596 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 24 % 2 2024-03-26T00:38:30.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-26T00:38:30.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj24 remove 2024-03-26T00:38:30.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:38:30.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:38:30.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:38:30.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:38:30.599 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:38:30.600 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj24 remove 2024-03-26T00:38:31.686 INFO:tasks.workunit.client.0.smithi129.stdout:remove 1#1:7eacede5:::obj24:head# 2024-03-26T00:38:32.231 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-26T00:38:32.232 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 25 % 2 2024-03-26T00:38:32.234 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-26T00:38:32.234 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj25 remove 2024-03-26T00:38:32.234 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:38:32.235 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:38:32.235 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-26T00:38:32.235 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:38:32.235 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:38:32.235 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj25 remove 2024-03-26T00:38:33.189 INFO:tasks.workunit.client.0.smithi129.stdout:remove 0#1:7109478f:::obj25:head# 2024-03-26T00:38:33.732 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-26T00:38:33.732 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 26 % 2 2024-03-26T00:38:33.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-26T00:38:33.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj26 remove 2024-03-26T00:38:33.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-26T00:38:33.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-26T00:38:33.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-26T00:38:33.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-26T00:38:33.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:38:33.735 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj26 remove 2024-03-26T00:38:34.598 INFO:tasks.workunit.client.0.smithi129.stdout:remove 1#1:f3c2593a:::obj26:head# 2024-03-26T00:38:35.137 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:766: TEST_repair_stats_ec: activate_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-26T00:38:35.137 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:38:35.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:38:35.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-26T00:38:35.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:38:35.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-26T00:38:35.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:38:35.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:38:35.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:38:35.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:38:35.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-26T00:38:35.138 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-26T00:38:35.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:38:35.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:38:35.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:38:35.141 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:38:35.141 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:38:35.141 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:38:35.141 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:38:35.141 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:38:35.141 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:38:35.141 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:38:35.141 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:38:35.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:38:35.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:38:35.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:38:35.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:38:35.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:38:35.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:38:35.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:38:35.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:38:35.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-26T00:38:35.142 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-26T00:38:35.143 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.1 2024-03-26T00:38:35.144 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-26T00:38:35.144 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-26T00:38:35.144 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-26T00:38:35.145 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-26T00:38:35.146 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:38:35.146 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:38:35.146 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:38:35.178 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:38:35.175+0000 7f98094817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:38:35.181 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:38:35.175+0000 7f98094817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:38:35.182 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:38:35.179+0000 7f98094817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:38:35.470 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-26T00:38:35.470 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:38:35.470 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-26T00:38:35.470 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:38:35.470 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:38:35.470 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:38:35.471 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-26T00:38:35.471 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:38:35.471 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:38:35.471 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:38:35.724 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:38:35.719+0000 7f98094817c0 -1 Falling back to public interface 2024-03-26T00:38:35.765 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:38:36.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:38:36.767 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:38:36.768 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-26T00:38:36.768 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:38:36.768 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:38:36.768 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:38:37.117 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:38:38.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:38:38.119 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:38:38.119 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-26T00:38:38.120 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:38:38.120 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:38:38.120 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:38:38.429 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:38:38.480 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:38:38.475+0000 7f98094817c0 -1 osd.1 23 log_to_monitors true 2024-03-26T00:38:39.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:38:39.431 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:38:39.431 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-26T00:38:39.432 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:38:39.432 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:38:39.432 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-26T00:38:40.037 INFO:tasks.workunit.client.0.smithi129.stdout:osd.1 up in weight 1 up_from 27 up_thru 22 down_at 24 last_clean_interval [10,23) [v2:127.0.0.1:6802/3675451013,v1:127.0.0.1:6803/3675451013] [v2:127.0.0.1:6804/3675451013,v1:127.0.0.1:6805/3675451013] exists,up add161d5-0034-4ce0-a147-8d72f0847aa5 2024-03-26T00:38:40.039 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:38:40.039 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:38:40.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:38:40.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:767: TEST_repair_stats_ec: activate_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-26T00:38:40.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-26T00:38:40.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-26T00:38:40.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-26T00:38:40.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-26T00:38:40.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-26T00:38:40.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-26T00:38:40.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-26T00:38:40.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-26T00:38:40.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-26T00:38:40.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-26T00:38:40.040 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-26T00:38:40.041 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-26T00:38:40.041 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-26T00:38:40.041 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-26T00:38:40.041 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-26T00:38:40.041 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-26T00:38:40.041 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-26T00:38:40.041 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-26T00:38:40.041 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:38:40.041 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:38:40.041 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:38:40.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' 2024-03-26T00:38:40.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-26T00:38:40.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-26T00:38:40.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-26T00:38:40.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-26T00:38:40.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-26T00:38:40.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-26T00:38:40.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-26T00:38:40.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-26T00:38:40.043 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-26T00:38:40.044 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-26T00:38:40.044 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-26T00:38:40.044 INFO:tasks.workunit.client.0.smithi129.stdout:start osd.0 2024-03-26T00:38:40.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0a9dcf66-19ea-412a-b2e8-98944a1fb783 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61636/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-26T00:38:40.045 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-26T00:38:40.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-26T00:38:40.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-26T00:38:40.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-26T00:38:40.045 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-26T00:38:40.081 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:38:40.075+0000 7f3ec90d97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:38:40.083 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:38:40.079+0000 7f3ec90d97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:38:40.085 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:38:40.079+0000 7f3ec90d97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-26T00:38:40.439 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-26T00:38:40.439 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-26T00:38:40.439 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-26T00:38:40.440 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-26T00:38:40.440 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-26T00:38:40.440 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:38:40.440 INFO:tasks.workunit.client.0.smithi129.stdout:0 2024-03-26T00:38:40.440 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-26T00:38:40.441 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:38:40.441 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:38:40.624 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:38:40.619+0000 7f3ec90d97c0 -1 Falling back to public interface 2024-03-26T00:38:40.757 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:38:41.759 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:38:41.759 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:38:41.759 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-26T00:38:41.759 INFO:tasks.workunit.client.0.smithi129.stdout:1 2024-03-26T00:38:41.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:38:41.760 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:38:42.069 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:38:43.071 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:38:43.071 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:38:43.072 INFO:tasks.workunit.client.0.smithi129.stdout:2 2024-03-26T00:38:43.072 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-26T00:38:43.072 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:38:43.072 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:38:43.397 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-26T00:38:43.507 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-26T00:38:43.503+0000 7f3ec90d97c0 -1 osd.0 21 log_to_monitors true 2024-03-26T00:38:44.399 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-26T00:38:44.399 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-26T00:38:44.399 INFO:tasks.workunit.client.0.smithi129.stdout:3 2024-03-26T00:38:44.400 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-26T00:38:44.400 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-26T00:38:44.400 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-26T00:38:44.855 INFO:tasks.workunit.client.0.smithi129.stdout:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6810/4064097197,v1:127.0.0.1:6811/4064097197] [v2:127.0.0.1:6812/4064097197,v1:127.0.0.1:6813/4064097197] exists,up 543d7e79-dd02-4143-b238-c09e1a037342 2024-03-26T00:38:44.856 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-26T00:38:44.856 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-26T00:38:44.856 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-26T00:38:44.856 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:768: TEST_repair_stats_ec: wait_for_clean 2024-03-26T00:38:44.857 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:38:44.857 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:38:44.857 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:38:44.857 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:38:44.857 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:38:44.858 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:38:44.858 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:38:44.858 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:38:44.858 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:38:44.968 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:38:44.968 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:38:44.968 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:38:44.968 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:38:44.968 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:38:44.968 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:38:45.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:38:45.343 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:38:45.343 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-26T00:38:45.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:38:45.344 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:38:45.344 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:38:45.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-03-26T00:38:45.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-03-26T00:38:45.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882' 2024-03-26T00:38:45.578 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:38:45.578 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:38:45.723 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964116995 2024-03-26T00:38:45.723 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964116995 2024-03-26T00:38:45.723 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-115964116995' 2024-03-26T00:38:45.723 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:38:45.723 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:38:45.940 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509460 2024-03-26T00:38:45.940 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509460 2024-03-26T00:38:45.940 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-115964116995 2-64424509460' 2024-03-26T00:38:45.940 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:38:45.940 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018882 2024-03-26T00:38:45.940 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:38:45.941 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:38:45.943 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018882 2024-03-26T00:38:45.943 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:38:45.943 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-03-26T00:38:45.944 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 128849018882 2024-03-26T00:38:45.944 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-03-26T00:38:45.944 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:38:46.258 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 128849018882 2024-03-26T00:38:46.258 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:38:47.260 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:38:47.261 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:38:47.582 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-03-26T00:38:47.607 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:38:47.608 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-115964116995 2024-03-26T00:38:47.608 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:38:47.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:38:47.608 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-115964116995 2024-03-26T00:38:47.608 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:38:47.608 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964116995 2024-03-26T00:38:47.609 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 115964116995' 2024-03-26T00:38:47.609 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 115964116995 2024-03-26T00:38:47.609 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:38:47.943 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964116995 -lt 115964116995 2024-03-26T00:38:47.944 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:38:47.944 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:38:47.944 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509460 2024-03-26T00:38:47.946 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:38:47.947 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509460 2024-03-26T00:38:47.947 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:38:47.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509460 2024-03-26T00:38:47.949 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-03-26T00:38:47.949 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 64424509460 2024-03-26T00:38:47.949 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:38:48.282 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509460 -lt 64424509460 2024-03-26T00:38:48.284 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:38:48.284 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:38:48.284 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:38:48.671 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-26T00:38:48.672 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:38:48.672 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:38:48.672 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:38:48.673 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:38:48.673 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:38:48.673 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:38:48.673 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:38:49.010 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-26T00:38:49.010 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:38:49.010 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:38:49.011 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:38:49.395 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-26T00:38:49.395 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:38:49.395 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:38:49.395 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:770: TEST_repair_stats_ec: repair 1.0 2024-03-26T00:38:49.396 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-03-26T00:38:49.396 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-03-26T00:38:49.396 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-26T00:38:49.396 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:38:49.396 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-26T00:38:49.397 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:38:49.702 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-26T00:37:30.855394+0000 2024-03-26T00:38:49.703 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-03-26T00:38:49.975 INFO:tasks.workunit.client.0.smithi129.stderr:instructing pg 1.0s0 on osd.1 to repair 2024-03-26T00:38:49.990 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-03-26T00:37:30.855394+0000 2024-03-26T00:38:49.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-03-26T00:38:49.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-26T00:37:30.855394+0000 2024-03-26T00:38:49.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-26T00:38:49.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-26T00:38:49.991 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:38:49.992 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-26T00:38:49.992 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-26T00:38:49.992 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:38:49.992 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:38:49.992 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-26T00:38:50.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:37:30.855394+0000 '>' 2024-03-26T00:37:30.855394+0000 2024-03-26T00:38:50.283 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:38:51.285 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:38:51.285 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:38:51.286 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-26T00:38:51.286 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-26T00:38:51.286 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:38:51.287 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:38:51.287 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-26T00:38:51.605 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:37:30.855394+0000 '>' 2024-03-26T00:37:30.855394+0000 2024-03-26T00:38:51.605 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:38:52.605 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:38:52.605 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:38:52.606 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-26T00:38:52.606 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-26T00:38:52.606 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:38:52.607 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:38:52.607 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-26T00:38:52.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:37:30.855394+0000 '>' 2024-03-26T00:37:30.855394+0000 2024-03-26T00:38:52.922 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:38:53.923 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:38:53.924 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:38:53.924 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-26T00:38:53.924 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-26T00:38:53.925 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:38:53.925 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:38:53.925 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-26T00:38:54.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:37:30.855394+0000 '>' 2024-03-26T00:37:30.855394+0000 2024-03-26T00:38:54.243 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-26T00:38:55.244 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-26T00:38:55.244 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-26T00:38:55.246 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-26T00:38:55.246 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-26T00:38:55.246 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-26T00:38:55.246 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-26T00:38:55.246 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-26T00:38:55.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-26T00:38:50.590779+0000 '>' 2024-03-26T00:37:30.855394+0000 2024-03-26T00:38:55.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-26T00:38:55.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:771: TEST_repair_stats_ec: wait_for_clean 2024-03-26T00:38:55.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-26T00:38:55.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-26T00:38:55.564 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-26T00:38:55.565 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-26T00:38:55.565 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-26T00:38:55.565 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-26T00:38:55.565 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-26T00:38:55.565 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-26T00:38:55.565 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-26T00:38:55.687 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-26T00:38:55.687 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-26T00:38:55.687 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-26T00:38:55.687 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-26T00:38:55.687 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:38:55.688 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:38:56.004 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:38:56.004 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:38:56.004 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-26T00:38:56.004 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:38:56.004 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:38:56.005 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:38:56.152 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018885 2024-03-26T00:38:56.152 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018885 2024-03-26T00:38:56.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018885' 2024-03-26T00:38:56.153 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:38:56.153 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:38:56.296 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964116998 2024-03-26T00:38:56.296 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964116998 2024-03-26T00:38:56.296 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018885 1-115964116998' 2024-03-26T00:38:56.296 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:38:56.296 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:38:56.439 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509463 2024-03-26T00:38:56.439 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509463 2024-03-26T00:38:56.439 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018885 1-115964116998 2-64424509463' 2024-03-26T00:38:56.439 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:38:56.440 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018885 2024-03-26T00:38:56.440 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:38:56.441 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:38:56.442 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018885 2024-03-26T00:38:56.442 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:38:56.443 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018885 2024-03-26T00:38:56.443 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 128849018885 2024-03-26T00:38:56.444 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018885' 2024-03-26T00:38:56.444 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:38:56.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018884 -lt 128849018885 2024-03-26T00:38:56.753 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:38:57.754 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:38:57.756 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:38:58.057 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018885 -lt 128849018885 2024-03-26T00:38:58.057 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:38:58.058 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-115964116998 2024-03-26T00:38:58.058 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:38:58.059 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:38:58.060 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-115964116998 2024-03-26T00:38:58.060 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:38:58.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964116998 2024-03-26T00:38:58.062 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 115964116998 2024-03-26T00:38:58.062 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 115964116998' 2024-03-26T00:38:58.063 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:38:58.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964116998 -lt 115964116998 2024-03-26T00:38:58.349 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:38:58.350 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509463 2024-03-26T00:38:58.350 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:38:58.352 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:38:58.353 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509463 2024-03-26T00:38:58.353 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:38:58.354 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509463 2024-03-26T00:38:58.354 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 64424509463 2024-03-26T00:38:58.355 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2024-03-26T00:38:58.355 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:38:58.707 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509463 -lt 64424509463 2024-03-26T00:38:58.707 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-26T00:38:58.708 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:38:58.708 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:38:59.120 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-26T00:38:59.120 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-26T00:38:59.121 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-26T00:38:59.121 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-26T00:38:59.121 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-26T00:38:59.121 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-26T00:38:59.122 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-26T00:38:59.122 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-26T00:38:59.428 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-26T00:38:59.429 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-26T00:38:59.429 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-26T00:38:59.429 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-26T00:38:59.808 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-26T00:38:59.809 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-26T00:38:59.809 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-26T00:38:59.809 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:772: TEST_repair_stats_ec: ceph pg dump pgs 2024-03-26T00:39:00.123 INFO:tasks.workunit.client.0.smithi129.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-26T00:39:00.123 INFO:tasks.workunit.client.0.smithi129.stdout:1.0 30 0 0 0 0 210 0 0 30 0 30 active+clean 2024-03-26T00:38:52.571437+0000 21'30 31:134 [1,0,2] 1 [1,0,2] 1 21'30 2024-03-26T00:38:50.590779+0000 21'30 2024-03-26T00:38:50.590779+0000 0 1 periodic scrub scheduled @ 2024-03-27T00:38:50.590779+0000 17 0 2024-03-26T00:39:00.123 INFO:tasks.workunit.client.0.smithi129.stdout: 2024-03-26T00:39:00.123 INFO:tasks.workunit.client.0.smithi129.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-26T00:39:00.124 INFO:tasks.workunit.client.0.smithi129.stderr:dumped pgs 2024-03-26T00:39:00.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:773: TEST_repair_stats_ec: flush_pg_stats 2024-03-26T00:39:00.139 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-26T00:39:00.140 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-26T00:39:00.475 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-26T00:39:00.475 INFO:tasks.workunit.client.0.smithi129.stderr:1 2024-03-26T00:39:00.475 INFO:tasks.workunit.client.0.smithi129.stderr:2' 2024-03-26T00:39:00.475 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-26T00:39:00.475 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:39:00.476 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-26T00:39:00.613 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018887 2024-03-26T00:39:00.614 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018887 2024-03-26T00:39:00.614 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018887' 2024-03-26T00:39:00.614 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:39:00.614 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-26T00:39:00.744 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964117000 2024-03-26T00:39:00.744 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964117000 2024-03-26T00:39:00.744 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018887 1-115964117000' 2024-03-26T00:39:00.744 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-26T00:39:00.745 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-26T00:39:00.877 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509465 2024-03-26T00:39:00.877 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509465 2024-03-26T00:39:00.878 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018887 1-115964117000 2-64424509465' 2024-03-26T00:39:00.878 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:39:00.878 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018887 2024-03-26T00:39:00.878 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:39:00.879 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-26T00:39:00.880 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018887 2024-03-26T00:39:00.880 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:39:00.881 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018887 2024-03-26T00:39:00.881 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018887' 2024-03-26T00:39:00.882 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.0 seq 128849018887 2024-03-26T00:39:00.882 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-26T00:39:01.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018887 -lt 128849018887 2024-03-26T00:39:01.182 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:39:01.183 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-115964117000 2024-03-26T00:39:01.183 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:39:01.186 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-26T00:39:01.187 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-115964117000 2024-03-26T00:39:01.187 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:39:01.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964117000 2024-03-26T00:39:01.189 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.1 seq 115964117000 2024-03-26T00:39:01.189 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 115964117000' 2024-03-26T00:39:01.189 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-26T00:39:01.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964117000 -lt 115964117000 2024-03-26T00:39:01.568 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-26T00:39:01.569 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509465 2024-03-26T00:39:01.569 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-26T00:39:01.570 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-26T00:39:01.572 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509465 2024-03-26T00:39:01.572 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-26T00:39:01.573 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509465 2024-03-26T00:39:01.573 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509465' 2024-03-26T00:39:01.574 INFO:tasks.workunit.client.0.smithi129.stdout:waiting osd.2 seq 64424509465 2024-03-26T00:39:01.574 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:39:01.893 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509464 -lt 64424509465 2024-03-26T00:39:01.894 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-26T00:39:02.895 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-26T00:39:02.896 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-26T00:39:03.216 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509465 -lt 64424509465 2024-03-26T00:39:03.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:776: TEST_repair_stats_ec: jq .info.stats.stat_sum 2024-03-26T00:39:03.217 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:776: TEST_repair_stats_ec: ceph pg 1.0 query 2024-03-26T00:39:03.433 INFO:tasks.workunit.client.0.smithi129.stdout:{ 2024-03-26T00:39:03.433 INFO:tasks.workunit.client.0.smithi129.stdout: "num_bytes": 210, 2024-03-26T00:39:03.433 INFO:tasks.workunit.client.0.smithi129.stdout: "num_objects": 30, 2024-03-26T00:39:03.433 INFO:tasks.workunit.client.0.smithi129.stdout: "num_object_clones": 0, 2024-03-26T00:39:03.433 INFO:tasks.workunit.client.0.smithi129.stdout: "num_object_copies": 90, 2024-03-26T00:39:03.433 INFO:tasks.workunit.client.0.smithi129.stdout: "num_objects_missing_on_primary": 0, 2024-03-26T00:39:03.433 INFO:tasks.workunit.client.0.smithi129.stdout: "num_objects_missing": 0, 2024-03-26T00:39:03.433 INFO:tasks.workunit.client.0.smithi129.stdout: "num_objects_degraded": 0, 2024-03-26T00:39:03.433 INFO:tasks.workunit.client.0.smithi129.stdout: "num_objects_misplaced": 0, 2024-03-26T00:39:03.434 INFO:tasks.workunit.client.0.smithi129.stdout: "num_objects_unfound": 0, 2024-03-26T00:39:03.434 INFO:tasks.workunit.client.0.smithi129.stdout: "num_objects_dirty": 30, 2024-03-26T00:39:03.434 INFO:tasks.workunit.client.0.smithi129.stdout: "num_whiteouts": 0, 2024-03-26T00:39:03.434 INFO:tasks.workunit.client.0.smithi129.stdout: "num_read": 0, 2024-03-26T00:39:03.434 INFO:tasks.workunit.client.0.smithi129.stdout: "num_read_kb": 0, 2024-03-26T00:39:03.434 INFO:tasks.workunit.client.0.smithi129.stdout: "num_write": 30, 2024-03-26T00:39:03.434 INFO:tasks.workunit.client.0.smithi129.stdout: "num_write_kb": 30, 2024-03-26T00:39:03.434 INFO:tasks.workunit.client.0.smithi129.stdout: "num_scrub_errors": 0, 2024-03-26T00:39:03.434 INFO:tasks.workunit.client.0.smithi129.stdout: "num_shallow_scrub_errors": 0, 2024-03-26T00:39:03.434 INFO:tasks.workunit.client.0.smithi129.stdout: "num_deep_scrub_errors": 0, 2024-03-26T00:39:03.434 INFO:tasks.workunit.client.0.smithi129.stdout: "num_objects_recovered": 26, 2024-03-26T00:39:03.434 INFO:tasks.workunit.client.0.smithi129.stdout: "num_bytes_recovered": 182, 2024-03-26T00:39:03.434 INFO:tasks.workunit.client.0.smithi129.stdout: "num_keys_recovered": 0, 2024-03-26T00:39:03.435 INFO:tasks.workunit.client.0.smithi129.stdout: "num_objects_omap": 0, 2024-03-26T00:39:03.435 INFO:tasks.workunit.client.0.smithi129.stdout: "num_objects_hit_set_archive": 0, 2024-03-26T00:39:03.435 INFO:tasks.workunit.client.0.smithi129.stdout: "num_bytes_hit_set_archive": 0, 2024-03-26T00:39:03.435 INFO:tasks.workunit.client.0.smithi129.stdout: "num_flush": 0, 2024-03-26T00:39:03.435 INFO:tasks.workunit.client.0.smithi129.stdout: "num_flush_kb": 0, 2024-03-26T00:39:03.435 INFO:tasks.workunit.client.0.smithi129.stdout: "num_evict": 0, 2024-03-26T00:39:03.435 INFO:tasks.workunit.client.0.smithi129.stdout: "num_evict_kb": 0, 2024-03-26T00:39:03.435 INFO:tasks.workunit.client.0.smithi129.stdout: "num_promote": 0, 2024-03-26T00:39:03.435 INFO:tasks.workunit.client.0.smithi129.stdout: "num_flush_mode_high": 0, 2024-03-26T00:39:03.435 INFO:tasks.workunit.client.0.smithi129.stdout: "num_flush_mode_low": 0, 2024-03-26T00:39:03.435 INFO:tasks.workunit.client.0.smithi129.stdout: "num_evict_mode_some": 0, 2024-03-26T00:39:03.435 INFO:tasks.workunit.client.0.smithi129.stdout: "num_evict_mode_full": 0, 2024-03-26T00:39:03.436 INFO:tasks.workunit.client.0.smithi129.stdout: "num_objects_pinned": 0, 2024-03-26T00:39:03.436 INFO:tasks.workunit.client.0.smithi129.stdout: "num_legacy_snapsets": 0, 2024-03-26T00:39:03.436 INFO:tasks.workunit.client.0.smithi129.stdout: "num_large_omap_objects": 0, 2024-03-26T00:39:03.436 INFO:tasks.workunit.client.0.smithi129.stdout: "num_objects_manifest": 0, 2024-03-26T00:39:03.436 INFO:tasks.workunit.client.0.smithi129.stdout: "num_omap_bytes": 0, 2024-03-26T00:39:03.436 INFO:tasks.workunit.client.0.smithi129.stdout: "num_omap_keys": 0, 2024-03-26T00:39:03.436 INFO:tasks.workunit.client.0.smithi129.stdout: "num_objects_repaired": 26 2024-03-26T00:39:03.436 INFO:tasks.workunit.client.0.smithi129.stdout:} 2024-03-26T00:39:03.436 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: ceph pg 1.0 query 2024-03-26T00:39:03.437 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: jq .info.stats.stat_sum.num_objects_repaired 2024-03-26T00:39:03.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: COUNT=26 2024-03-26T00:39:03.598 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:778: TEST_repair_stats_ec: test 26 = 26 2024-03-26T00:39:03.599 INFO:tasks.workunit.client.0.smithi129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: expr 3 - 1 2024-03-26T00:39:03.600 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: seq 0 2 2024-03-26T00:39:03.601 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-26T00:39:03.601 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:781: TEST_repair_stats_ec: '[' 0 = 0 -o 0 = 1 ']' 2024-03-26T00:39:03.602 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: expr 26 / 2 2024-03-26T00:39:03.602 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: repair=13 2024-03-26T00:39:03.603 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-03-26T00:39:03.603 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 0 )' 2024-03-26T00:39:03.894 INFO:tasks.workunit.client.0.smithi129.stderr:dumped all 2024-03-26T00:39:03.916 INFO:tasks.workunit.client.0.smithi129.stdout:{ 2024-03-26T00:39:03.916 INFO:tasks.workunit.client.0.smithi129.stdout: "osd": 0, 2024-03-26T00:39:03.916 INFO:tasks.workunit.client.0.smithi129.stdout: "up_from": 30, 2024-03-26T00:39:03.916 INFO:tasks.workunit.client.0.smithi129.stdout: "seq": 128849018887, 2024-03-26T00:39:03.917 INFO:tasks.workunit.client.0.smithi129.stdout: "num_pgs": 1, 2024-03-26T00:39:03.917 INFO:tasks.workunit.client.0.smithi129.stdout: "num_osds": 1, 2024-03-26T00:39:03.917 INFO:tasks.workunit.client.0.smithi129.stdout: "num_per_pool_osds": 1, 2024-03-26T00:39:03.917 INFO:tasks.workunit.client.0.smithi129.stdout: "num_per_pool_omap_osds": 1, 2024-03-26T00:39:03.917 INFO:tasks.workunit.client.0.smithi129.stdout: "kb": 104857600, 2024-03-26T00:39:03.917 INFO:tasks.workunit.client.0.smithi129.stdout: "kb_used": 28480, 2024-03-26T00:39:03.917 INFO:tasks.workunit.client.0.smithi129.stdout: "kb_used_data": 376, 2024-03-26T00:39:03.917 INFO:tasks.workunit.client.0.smithi129.stdout: "kb_used_omap": 4, 2024-03-26T00:39:03.917 INFO:tasks.workunit.client.0.smithi129.stdout: "kb_used_meta": 28091, 2024-03-26T00:39:03.917 INFO:tasks.workunit.client.0.smithi129.stdout: "kb_avail": 104829120, 2024-03-26T00:39:03.917 INFO:tasks.workunit.client.0.smithi129.stdout: "statfs": { 2024-03-26T00:39:03.917 INFO:tasks.workunit.client.0.smithi129.stdout: "total": 107374182400, 2024-03-26T00:39:03.917 INFO:tasks.workunit.client.0.smithi129.stdout: "available": 107345018880, 2024-03-26T00:39:03.917 INFO:tasks.workunit.client.0.smithi129.stdout: "internally_reserved": 0, 2024-03-26T00:39:03.918 INFO:tasks.workunit.client.0.smithi129.stdout: "allocated": 385024, 2024-03-26T00:39:03.918 INFO:tasks.workunit.client.0.smithi129.stdout: "data_stored": 206308, 2024-03-26T00:39:03.918 INFO:tasks.workunit.client.0.smithi129.stdout: "data_compressed": 0, 2024-03-26T00:39:03.918 INFO:tasks.workunit.client.0.smithi129.stdout: "data_compressed_allocated": 0, 2024-03-26T00:39:03.918 INFO:tasks.workunit.client.0.smithi129.stdout: "data_compressed_original": 0, 2024-03-26T00:39:03.918 INFO:tasks.workunit.client.0.smithi129.stdout: "omap_allocated": 4402, 2024-03-26T00:39:03.918 INFO:tasks.workunit.client.0.smithi129.stdout: "internal_metadata": 28765902 2024-03-26T00:39:03.918 INFO:tasks.workunit.client.0.smithi129.stdout: }, 2024-03-26T00:39:03.918 INFO:tasks.workunit.client.0.smithi129.stdout: "hb_peers": [ 2024-03-26T00:39:03.918 INFO:tasks.workunit.client.0.smithi129.stdout: 1, 2024-03-26T00:39:03.918 INFO:tasks.workunit.client.0.smithi129.stdout: 2 2024-03-26T00:39:03.918 INFO:tasks.workunit.client.0.smithi129.stdout: ], 2024-03-26T00:39:03.918 INFO:tasks.workunit.client.0.smithi129.stdout: "snap_trim_queue_len": 0, 2024-03-26T00:39:03.919 INFO:tasks.workunit.client.0.smithi129.stdout: "num_snap_trimming": 0, 2024-03-26T00:39:03.919 INFO:tasks.workunit.client.0.smithi129.stdout: "num_shards_repaired": 26, 2024-03-26T00:39:03.919 INFO:tasks.workunit.client.0.smithi129.stdout: "op_queue_age_hist": { 2024-03-26T00:39:03.919 INFO:tasks.workunit.client.0.smithi129.stdout: "histogram": [], 2024-03-26T00:39:03.919 INFO:tasks.workunit.client.0.smithi129.stdout: "upper_bound": 1 2024-03-26T00:39:03.919 INFO:tasks.workunit.client.0.smithi129.stdout: }, 2024-03-26T00:39:03.919 INFO:tasks.workunit.client.0.smithi129.stdout: "perf_stat": { 2024-03-26T00:39:03.920 INFO:tasks.workunit.client.0.smithi129.stdout: "commit_latency_ms": 0, 2024-03-26T00:39:03.920 INFO:tasks.workunit.client.0.smithi129.stdout: "apply_latency_ms": 0, 2024-03-26T00:39:03.920 INFO:tasks.workunit.client.0.smithi129.stdout: "commit_latency_ns": 0, 2024-03-26T00:39:03.920 INFO:tasks.workunit.client.0.smithi129.stdout: "apply_latency_ns": 0 2024-03-26T00:39:03.920 INFO:tasks.workunit.client.0.smithi129.stdout: }, 2024-03-26T00:39:03.920 INFO:tasks.workunit.client.0.smithi129.stdout: "alerts": [] 2024-03-26T00:39:03.920 INFO:tasks.workunit.client.0.smithi129.stdout:} 2024-03-26T00:39:03.920 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-03-26T00:39:03.920 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 0 ).num_shards_repaired' 2024-03-26T00:39:04.205 INFO:tasks.workunit.client.0.smithi129.stderr:dumped all 2024-03-26T00:39:04.222 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: COUNT=26 2024-03-26T00:39:04.222 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: test 26 = 13 2024-03-26T00:39:04.222 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: return 1 2024-03-26T00:39:04.222 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: return 1 2024-03-26T00:39:04.222 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: code=1 2024-03-26T00:39:04.222 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-scrub-repair 1 2024-03-26T00:39:04.223 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-26T00:39:04.223 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=1 2024-03-26T00:39:04.223 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-26T00:39:04.223 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-26T00:39:04.223 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-26T00:39:04.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-26T00:39:04.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-26T00:39:04.225 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-26T00:39:04.335 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-26T00:39:04.335 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-26T00:39:04.336 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-26T00:39:04.336 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-26T00:39:04.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-26T00:39:04.337 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-26T00:39:04.338 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-26T00:39:04.338 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:39:04.338 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-26T00:39:04.339 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-26T00:39:04.339 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T00:39:04.340 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-26T00:39:04.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 1 = 1 ']' 2024-03-26T00:39:04.341 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: '[' -n '' ']' 2024-03-26T00:39:04.342 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:194: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2024-03-26T00:39:04.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:195: teardown: mv td/osd-scrub-repair/mgr.x.log td/osd-scrub-repair/mon.a.log td/osd-scrub-repair/osd.0.log td/osd-scrub-repair/osd.1.log td/osd-scrub-repair/osd.2.log /home/ubuntu/cephtest/archive/log 2024-03-26T00:39:04.343 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-26T00:39:04.353 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-26T00:39:04.353 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-26T00:39:04.353 INFO:tasks.workunit.client.0.smithi129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61636 2024-03-26T00:39:04.353 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61636 2024-03-26T00:39:04.354 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-26T00:39:04.354 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-26T00:39:04.354 INFO:tasks.workunit.client.0.smithi129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 1 2024-03-26T00:39:04.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T00:39:04.356 INFO:tasks.workunit:Stopping ['scrub'] on client.0... 2024-03-26T00:39:04.356 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-26T00:39:04.670 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_4354a20109fd25703f6611a1a8f196608a256149/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_4354a20109fd25703f6611a1a8f196608a256149/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_4354a20109fd25703f6611a1a8f196608a256149/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi129 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4354a20109fd25703f6611a1a8f196608a256149 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh' 2024-03-26T00:39:04.902 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=9c3a80220d00416585562cb52d07b0a6 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_4354a20109fd25703f6611a1a8f196608a256149/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_4354a20109fd25703f6611a1a8f196608a256149/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_4354a20109fd25703f6611a1a8f196608a256149/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi129 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4354a20109fd25703f6611a1a8f196608a256149 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh' 2024-03-26T00:39:04.905 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-26T00:39:04.915 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-26T00:39:04.915 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-26T00:39:04.957 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-26T00:39:04.966 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-26T00:39:04.967 DEBUG:teuthology.orchestra.run.smithi129:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T00:39:04.987 INFO:teuthology.orchestra.run.smithi129.stdout: remote refid st t when poll reach delay offset jitter 2024-03-26T00:39:04.987 INFO:teuthology.orchestra.run.smithi129.stdout:============================================================================== 2024-03-26T00:39:04.988 INFO:teuthology.orchestra.run.smithi129.stdout:*hv01.front.sepi 44.4.53.2 2 u 21 128 377 0.085 -1.146 0.191 2024-03-26T00:39:04.988 INFO:teuthology.orchestra.run.smithi129.stdout:+hv02.front.sepi 63.231.80.2 3 u 19 128 377 0.084 -0.106 0.284 2024-03-26T00:39:04.988 INFO:teuthology.orchestra.run.smithi129.stdout:+hv03.front.sepi 74.6.168.72 3 u 85 128 377 0.129 -0.637 0.269 2024-03-26T00:39:04.988 INFO:teuthology.orchestra.run.smithi129.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-03-26T00:39:04.989 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-26T00:39:04.998 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-26T00:39:04.998 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-26T00:39:05.007 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-26T00:39:05.016 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-26T00:39:05.024 INFO:teuthology.task.internal:Duration was 6479.515748 seconds 2024-03-26T00:39:05.024 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-26T00:39:05.033 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-26T00:39:05.033 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T00:39:05.070 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-26T00:39:05.070 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi129.front.sepia.ceph.com 2024-03-26T00:39:05.070 DEBUG:teuthology.orchestra.run.smithi129:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T00:39:05.122 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-26T00:39:05.123 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T00:39:05.207 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-26T00:39:05.208 DEBUG:teuthology.orchestra.run.smithi129:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T00:39:05.302 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-26T00:39:05.312 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-26T00:39:05.313 DEBUG:teuthology.orchestra.run.smithi129:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T00:39:05.326 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-26T00:39:05.339 DEBUG:teuthology.orchestra.run.smithi129:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T00:39:05.379 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern = core 2024-03-26T00:39:05.404 DEBUG:teuthology.orchestra.run.smithi129:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T00:39:05.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T00:39:05.450 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-26T00:39:05.464 INFO:teuthology.task.internal:Transferring archived files... 2024-03-26T00:39:05.466 DEBUG:teuthology.misc:Transferring archived files from smithi129:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-03-25_20:36:27-rados-wip-yuri3-testing-2024-03-25-0744-distro-default-smithi/7621635/remote/smithi129 2024-03-26T00:39:05.467 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T00:39:06.570 INFO:teuthology.task.internal:Removing archive directory... 2024-03-26T00:39:06.571 DEBUG:teuthology.orchestra.run.smithi129:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T00:39:06.616 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-26T00:39:06.627 INFO:teuthology.task.internal:Not uploading archives. 2024-03-26T00:39:06.627 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-26T00:39:06.637 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-26T00:39:06.637 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T00:39:06.665 INFO:teuthology.orchestra.run.smithi129.stdout: 658615 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 26 00:39 /home/ubuntu/cephtest 2024-03-26T00:39:06.665 INFO:teuthology.orchestra.run.smithi129.stdout: 671696 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Mar 25 22:59 /home/ubuntu/cephtest/mnt.0 2024-03-26T00:39:06.666 INFO:teuthology.orchestra.run.smithi129.stdout: 673722 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Mar 25 23:11 /home/ubuntu/cephtest/mnt.0/client.0 2024-03-26T00:39:06.666 INFO:teuthology.orchestra.run.smithi129.stdout: 803044 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Mar 25 23:11 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-26T00:39:06.666 INFO:teuthology.orchestra.run.smithi129.stdout: 803045 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Mar 26 00:39 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2024-03-26T00:39:06.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T00:39:06.667 INFO:teuthology.orchestra.run.smithi129.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-03-26T00:39:06.668 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi129 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-03-26T00:39:06.674 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-26T00:39:06.687 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-26T00:39:06.698 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi129 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-03-26T00:39:06.698 INFO:teuthology.nuke:Checking targets against current locks 2024-03-26T00:39:06.727 DEBUG:teuthology.nuke:shortname: smithi129 2024-03-26T00:39:06.728 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T00:39:06.755 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi129.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-25_20:36:27-rados-wip-yuri3-testing-2024-03-25-0744-distro-default-smithi/7621635', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 22:42:15.941866', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFcf1ZeqmvRJQbHCZDCLvapqnSlttyxH2G0hT6jN847ksNewhuhchW6nOj65DHNYawsvGdEjmdGdTeO2qwfXlvw='} 2024-03-26T00:39:06.809 INFO:teuthology.orchestra.console.smithi129:Power off 2024-03-26T00:39:06.809 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T00:39:06.836 DEBUG:teuthology.orchestra.console.smithi129:power off output: Chassis Power Control: Down/Off 2024-03-26T00:39:06.836 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T00:39:06.861 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is on 2024-03-26T00:39:10.863 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T00:39:10.975 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is on 2024-03-26T00:39:14.978 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T00:39:15.090 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is on 2024-03-26T00:39:19.095 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T00:39:19.205 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is off 2024-03-26T00:39:19.306 INFO:teuthology.orchestra.console.smithi129:Power off completed 2024-03-26T00:39:19.411 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/scrub} duration: 6479.515748023987 failure_reason: 'Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi129 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4354a20109fd25703f6611a1a8f196608a256149 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=9c3a80220d00416585562cb52d07b0a6 status: fail success: false 2024-03-26T00:39:19.412 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-26T00:39:19.492 INFO:teuthology.run:FAIL